Nov 03 19:36:58.160114 ip-10-0-142-244 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 19:36:58.160125 ip-10-0-142-244 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 19:36:58.160132 ip-10-0-142-244 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 19:36:58.160367 ip-10-0-142-244 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 19:37:08.178047 ip-10-0-142-244 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 19:37:08.178065 ip-10-0-142-244 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1681ef2097c941d0ab8e03c129ec1930 --
Nov 03 19:38:54.839210 ip-10-0-142-244 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:38:55.223873 ip-10-0-142-244 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:38:55.223873 ip-10-0-142-244 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:38:55.223873 ip-10-0-142-244 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:38:55.223873 ip-10-0-142-244 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:38:55.230160 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230076    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.230160 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230160    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.230160 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230163    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230167    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230170    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230173    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230176    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230179    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230187    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230190    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230193    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230196    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230199    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230202    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230204    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230207    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230210    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230213    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230215    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230218    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230220    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.230250 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230223    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230225    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230228    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230230    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230233    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230235    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230238    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230240    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230243    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230245    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230249    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230251    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230254    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230257    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230260    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230263    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230266    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230269    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230271    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230274    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.230684 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230277    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230279    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230282    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230284    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230286    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230289    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230291    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230294    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230297    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230299    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230302    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230304    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230307    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230312    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230317    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230320    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230323    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230326    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230328    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.231161 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230331    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230334    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230337    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230340    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230343    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230346    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230350    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230353    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230355    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230358    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230361    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230363    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230366    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230368    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230371    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230375    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230379    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230382    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230385    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.231605 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230387    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230390    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230394    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230396    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230399    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230402    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230405    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230408    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230411    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230414    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230416    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230419    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230421    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230423    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230426    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230429    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230431    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230434    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230437    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230439    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.232068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230442    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230445    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230448    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230452    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230454    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230457    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.230459    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231271    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231278    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231280    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231283    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231286    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231289    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231292    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231295    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231298    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231301    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231303    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231306    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.232529 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231309    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231311    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231314    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231317    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231319    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231322    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231324    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231327    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231330    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231332    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231335    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231337    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231340    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231344    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231348    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231351    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231355    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231358    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231360    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.233099 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231363    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231366    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231368    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231371    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231374    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231377    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231379    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231382    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231385    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231388    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231391    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231393    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231396    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231399    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231401    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231404    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231408    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231413    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231415    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231418    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.233546 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231420    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231423    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231425    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231428    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231430    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231434    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231437    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231439    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231442    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231445    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231448    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231450    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231453    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231455    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231458    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231460    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231463    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231466    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231469    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231472    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.234245 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231475    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231477    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231480    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231482    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231485    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231488    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231491    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231493    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231496    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231498    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231501    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231503    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231506    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231508    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231511    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231529    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231533    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231537    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231540    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.234804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231543    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231546    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231548    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231555    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231557    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231560    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231563    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231565    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231568    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231570    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231573    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231576    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231578    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231581    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231584    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.231586    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232083    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232088    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232092    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232095    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.235268 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232097    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232101    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232103    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232106    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232109    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232111    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232114    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232117    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232119    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232122    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232124    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232127    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232130    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232133    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232136    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232139    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232142    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232144    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232147    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232149    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.235841 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232152    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232154    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232157    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232160    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232163    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232165    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232168    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232172    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232175    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232178    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232181    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232184    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232186    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232189    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232192    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232195    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232198    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232200    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232203    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232205    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.236370 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232222    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232226    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232229    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232232    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232234    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232238    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232240    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232243    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232246    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232250    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232252    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232255    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232258    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232263    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232266    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232269    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232271    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232274    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232276    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.236883 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232279    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232282    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232284    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232287    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232289    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232292    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232294    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232297    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232299    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232302    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232304    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232307    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232309    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232312    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232314    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232317    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232319    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232322    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232324    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.237347 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232328    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232331    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232334    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232336    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232339    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232342    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232345    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232347    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232350    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232354    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232358    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232361    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232363    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232366    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232368    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232371    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232373    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232376    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232379    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.237804 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232381    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232384    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232386    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232389    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.232393    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233472    2585 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233483    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233490    2585 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233495    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233500    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233504    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233508    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233512    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233516    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233519    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233523    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233527    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233530    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233533    2585 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233536    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233539    2585 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233542    2585 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233545    2585 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233550    2585 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:38:55.238304 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233553    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233556    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233559    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233563    2585 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233567    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233570    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233573    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233576    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233579    2585 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233582    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233585    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233589    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233592    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233596    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233599    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233602    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233605    2585 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233608    2585 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.233611    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235060    2585 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235069    2585 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235073    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235077    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235084    2585 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:38:55.238880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235090    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235093    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235096    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235100    2585 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235103    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235106    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235110    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235113    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235116    2585 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235119    2585 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235123    2585 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235127    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235131    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235134    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235137    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235141    2585 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235144    2585 flags.go:64] FLAG: --help="false"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235147    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235151    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235154    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235157    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235160    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235164    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:38:55.239508 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235167    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235170    2585 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235173    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235176    2585 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235180    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235185    2585 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235188    2585 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235191    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235195    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235198    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235201    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235204    2585 flags.go:64] FLAG: --lock-file=""
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235207    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235210    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235213    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235219    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235222    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235226    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235229    2585 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235232    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235236    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235238    2585 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235241    2585 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235247    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235250    2585 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:38:55.240063 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235261    2585 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235264    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235267    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235270    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235274    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235277    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235280    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235283    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235293    2585 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235296    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235299    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235303    2585 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235307    2585 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235310    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235313    2585 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235317    2585 flags.go:64] FLAG: --port="10250"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235320    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235323    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03af00eb5275d65bf"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235326    2585 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235329    2585 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235332    2585 flags.go:64] FLAG: --register-node="true"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235336    2585 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235340    2585 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235343    2585 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235345    2585 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235348    2585 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:38:55.240652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235352    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235355    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235358    2585 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235361    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235365    2585 flags.go:64] FLAG: --runonce="false"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235368    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235372    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235375    2585 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235378    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235380    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235384    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235387    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235390    2585 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235393    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235396    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235399    2585 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235402    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235405    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235408    2585 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235411    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235418    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235421    2585 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235424    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235429    2585 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235432    2585 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:38:55.241287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235435    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235438    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235442    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235445    2585 flags.go:64] FLAG: --v="2"
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235449    2585 flags.go:64] FLAG: --version="false"
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235454    2585 flags.go:64] FLAG: --vmodule=""
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235458    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235461    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235599    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235603    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235606    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235610    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235613    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235616    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235619    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235622    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235624    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235627    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235629    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235632    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235635    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235637    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.241879 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235640    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235659    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235662    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235665    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235669    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235671    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235674    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235677    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235681    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235684    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235686    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235689    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235692    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235695    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235698    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235701    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235704    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235707    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235709    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235712    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.242415 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235714    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235717    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235720    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235723    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235726    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235728    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235731    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235734    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235736    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235739    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235741    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235744    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235746    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235749    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235751    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235754    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235756    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235759    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235762    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235764    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.242931 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235767    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235769    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235772    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235775    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235779    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235782    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235785    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235788    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235790    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235793    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235795    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235798    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235801    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235805    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235807    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235810    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235819    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235822    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235825    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.243396 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235828    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235830    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235833    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235835    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235838    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235840    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235843    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235846    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235848    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235851    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235856    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235870    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235873    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235876    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235878    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235881    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235884    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235888    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235892    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.243833 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235895    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235898    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235900    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235903    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235905    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235908    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235911    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235915    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235917    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235920    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235922    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235925    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235928    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.235931    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.235959    2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236001    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.244301 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236006    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236009    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236012    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236015    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236018    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236020    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236023    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236027    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236029    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236032    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236035    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236037    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236040    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236042    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236045    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236047    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236050    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236052    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236055    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236058    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236060    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.244681 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236063    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236066    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236068    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236072    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236075    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236078    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236080    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236083    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236086    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236089    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236092    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236094    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236097    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236100    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236102    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236105    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236107    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236110    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236113    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236116    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.245190 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236119    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236122    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236126    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236128    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236131    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236134    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236136    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236139    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236142    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236144    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236147    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236149    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236152    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236154    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236157    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236160    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236163    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236167    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236171    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.245645 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236174    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236179    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236181    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236184    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236186    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236189    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236192    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236194    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236197    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236199    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236202    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236206    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236209    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236211    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236214    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236217    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236219    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236221    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236224    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.246102 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236227    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236229    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236232    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236235    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236237    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236240    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236242    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236245    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236248    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236252    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236254    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236257    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236262    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236266    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236269    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236273    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236275    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236278    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.246549 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236281    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236283    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236286    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236288    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236290    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236293    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236297    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.236299    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.244249    2585 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.244265    2585 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.245021    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.245028    2585 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245087    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245093    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245097    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245100    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245103    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245106    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.247068 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245109    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245113    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245117    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245121    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245124    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245127    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245130    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245133    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245135    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245138    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245140    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245144    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245147    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245150    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245152    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245155    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245158    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245160    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245163    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.247484 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245165    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245168    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245171    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245173    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245176    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245179    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245182    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245185    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245187    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245190    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245193    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245198    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245201    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245204    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245207    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245210    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245213    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245216    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245218    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.247934 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245221    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245223    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245226    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245228    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245231    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245233    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245236    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245239    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245242    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245245    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245248    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245250    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245253    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245256    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245259    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245261    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245264    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245267    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245269    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.248407 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245272    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245274    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245277    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245280    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245282    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245285    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245287    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245290    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245293    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245295    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245298    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245301    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245303    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245306    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245308    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245311    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245314    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245317    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245319    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245322    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.248871 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245325    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245328    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245330    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245333    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245335    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245338    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245341    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245343    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245346    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245348    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245351    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245354    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245357    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245359    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245362    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245364    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245367    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245369    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245372    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245375    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.249349 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245377    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245380    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245382    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.245409    2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245564    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245569    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245573    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245575    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245578    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245581    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245583    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245586    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245588    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245591    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245594    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245596    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.249816 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245599    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245603    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245607    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245610    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245613    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245616    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245619    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245622    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245625    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245627    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245630    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245632    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245635    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245637    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245639    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245642    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245645    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245647    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245650    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.250213 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245652    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245656    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245658    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245661    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245664    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245666    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245669    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245672    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245674    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245677    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245679    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245682    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245685    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245688    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245690    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245693    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245695    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245698    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245700    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245703    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.250643 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245706    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245708    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245710    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245713    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245715    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245718    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245720    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245723    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245726    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245728    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245731    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245733    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245736    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245739    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245741    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245744    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245746    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245749    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245751    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245754    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.251122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245756    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245759    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245761    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245764    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245767    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245769    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245772    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245774    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245777    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245780    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245784    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245786    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245789    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245791    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245794    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245797    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245799    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245802    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245804    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.251620 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245807    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245810    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245812    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245815    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245817    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245820    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245822    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245825    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245827    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245830    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245832    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245835    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245838    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245840    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245843    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245845    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.252103 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.245886    2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245919    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245925    2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245928    2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245931    2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245934    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245937    2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245939    2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245942    2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245944    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245948    2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245952    2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245956    2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245958    2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245961    2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245964    2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245966    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245969    2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245971    2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:38:55.252471 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245974    2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245976    2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245979    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245981    2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245984    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245986    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245989    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245991    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245994    2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245996    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.245999    2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246001    2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246004    2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246006    2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246009    2585 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246012    2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246015    2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246017    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246020    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246023    2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:38:55.252956 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246025    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246028    2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246030    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246033    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246035    2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246038    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246040    2585 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246043    2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246045    2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246048    2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246050    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246053    2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246055    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246058    2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246060    2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246063    2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246065    2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246068    2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246070    2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246073    2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:38:55.253418 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246075    2585 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246078    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246080    2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246083    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246085    2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246088    2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246090    2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246093    2585 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246096    2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246099    2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246101    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246104    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246110    2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246113    2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246116    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246118    2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246121    2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246123    2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246127    2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:38:55.254030 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246131    2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246134    2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246137    2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246140    2585 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246143    2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246145    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246148    2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246150    2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246153    2585 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246156    2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246158    2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246161    2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246163    2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246166    2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246168    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246171    2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246173    2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246176    2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246180    2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:38:55.254477 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246183    2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246185    2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246189    2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246192    2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246195    2585 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246197    2585 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246201    2585 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246204    2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246207    2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:55.246209    2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.246259    2585 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.248663    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.249728    2585 server.go:1014] "Starting client certificate rotation"
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.249826    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.249883    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.252588    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:38:55.254965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.253703    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:38:55.266121 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.266104    2585 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:38:55.272719 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.272700    2585 log.go:25] "Validated CRI v1 image API"
Nov 03 19:38:55.273560 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.273529    2585 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:38:55.279110 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.279084    2585 fs.go:135] Filesystem UUIDs: map[54fb9646-e257-4ff7-a506-e4af69766d17:/dev/nvme0n1p4 587d2e0a-137f-4ab4-ab15-1322921743b1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 19:38:55.279192 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.279107    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 19:38:55.282246 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.282223    2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:38:55.284409 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.284300    2585 manager.go:217] Machine: {Timestamp:2026-11-03 19:38:55.282829125 +0000 UTC m=+0.348265640 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3136505 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ebf778b1c67e43996e7c5488401b1 SystemUUID:ec2ebf77-8b1c-67e4-3996-e7c5488401b1 BootID:1681ef20-97c9-41d0-ab8e-03c129ec1930 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:76:30:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:76:30:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:b9:88:b9:1d:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:38:55.285025 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.285013    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:38:55.285138 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.285119    2585 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:38:55.287941 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.287920    2585 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:38:55.288071 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.287942    2585 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-244.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:38:55.288683 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.288674    2585 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:38:55.288718 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.288685    2585 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:38:55.288718 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.288713    2585 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:38:55.288768 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.288726    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:38:55.290018 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.290007    2585 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:38:55.294690 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.294679    2585 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:38:55.294803 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.294794    2585 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:38:55.294840 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.294824    2585 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:38:55.294840 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.294834    2585 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:38:55.294934 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.294847    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:38:55.294934 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.294856    2585 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:38:55.294934 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.294875    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:38:55.295195 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.295184    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:38:55.297732 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.297717    2585 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:38:55.298417 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.298395    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fbnlt"
Nov 03 19:38:55.305166 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.305151    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:38:55.306022 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306006    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fbnlt"
Nov 03 19:38:55.306074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306010    2585 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:38:55.306074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306052    2585 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:38:55.306238 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306218    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:38:55.306271 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306243    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:38:55.306271 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306250    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:38:55.306271 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306256    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:38:55.306271 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306262    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:38:55.306271 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306268    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:38:55.306924 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306915    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:38:55.306960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306926    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:38:55.306960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306934    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:38:55.306960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306940    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:38:55.306960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306949    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:38:55.306960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.306958    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:38:55.307742 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.307732    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:38:55.307776 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.307746    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:38:55.312582 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.312558    2585 server.go:1263] "Started kubelet"
Nov 03 19:38:55.312664 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.312601    2585 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:38:55.312715 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.312669    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:38:55.312747 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.312735    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:38:55.313530 ip-10-0-142-244 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:38:55.314911 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.314892    2585 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:38:55.315841 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.315830    2585 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:38:55.320471 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.320448    2585 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 19:38:55.322782 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.322766    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:38:55.324066 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.324051    2585 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:38:55.324158 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.324144    2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-244.us-east-2.compute.internal\" not found"
Nov 03 19:38:55.324364 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.324345    2585 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:38:55.324803 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.324777    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:38:55.324803 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.324780    2585 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:38:55.324938 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.324818    2585 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:38:55.325035 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325014    2585 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:38:55.325035 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325033    2585 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:38:55.325131 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325058    2585 factory.go:153] Registering CRI-O factory
Nov 03 19:38:55.325131 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325071    2585 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:38:55.325131 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325112    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:38:55.325131 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325118    2585 factory.go:55] Registering systemd factory
Nov 03 19:38:55.325131 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325122    2585 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:38:55.325131 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325133    2585 factory.go:103] Registering Raw factory
Nov 03 19:38:55.325301 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.325142    2585 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:38:55.326340 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.326185    2585 manager.go:319] Starting recovery of all containers
Nov 03 19:38:55.326953 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.326936    2585 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-244.us-east-2.compute.internal" not found
Nov 03 19:38:55.327880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.327476    2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:38:55.327880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.327564    2585 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:38:55.329144 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.329125    2585 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-244.us-east-2.compute.internal\" not found" node="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.337836 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.337822    2585 manager.go:324] Recovery completed
Nov 03 19:38:55.340367 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.340346    2585 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-244.us-east-2.compute.internal" not found
Nov 03 19:38:55.344101 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.344087    2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:38:55.346246 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.346229    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:38:55.346311 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.346257    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:38:55.346311 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.346268    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:38:55.346829 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.346816    2585 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:38:55.346829 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.346827    2585 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:38:55.346913 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.346843    2585 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:38:55.346913 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.346847    2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:38:55.347595 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.347576    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:38:55.347664 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.347605    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:38:55.347664 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.347615    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:38:55.349066 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.349051    2585 policy_none.go:50] "Start"
Nov 03 19:38:55.349121 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.349072    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:38:55.349121 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.349087    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:38:55.351157 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.351144    2585 policy_none.go:44] "Start"
Nov 03 19:38:55.415841 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.397244    2585 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-142-244.us-east-2.compute.internal" not found
Nov 03 19:38:55.415841 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.413566    2585 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:38:55.415841 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.415074    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.416439    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.416458    2585 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.416477    2585 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.416506    2585 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.416763    2585 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.416810    2585 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.416822    2585 server.go:87] "Starting device plugin registration server"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.417123    2585 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.417144    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.417305    2585 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.417375    2585 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.417384    2585 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.418218    2585 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.418268    2585 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-244.us-east-2.compute.internal\" not found"
Nov 03 19:38:55.420074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.419983    2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:38:55.516971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.516896    2585 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"]
Nov 03 19:38:55.517098 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.516980    2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:38:55.517335 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.517322    2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:38:55.518045 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.518022    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:38:55.518137 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.518061    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:38:55.518137 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.518076    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:38:55.518137 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.518104    2585 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.518238 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.518061    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:38:55.518238 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.518181    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:38:55.518238 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.518191    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:38:55.520383 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.520371    2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:38:55.520523 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.520511    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.520563 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.520538    2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:38:55.521033 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.521019    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:38:55.521033 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.521028    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:38:55.521155 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.521041    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:38:55.521155 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.521045    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:38:55.521155 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.521051    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:38:55.521155 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.521061    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:38:55.523141 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.523128    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.523214 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.523153    2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:38:55.523825 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.523809    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:38:55.523901 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.523842    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:38:55.523901 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.523857    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:38:55.527425 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.527410    2585 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.527518 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.527433    2585 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-244.us-east-2.compute.internal\": node \"ip-10-0-142-244.us-east-2.compute.internal\" not found"
Nov 03 19:38:55.549464 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.549444    2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.555811 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.555797    2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.562497 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.562482    2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:38:55.568587 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.568573    2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:38:55.625083 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.625054    2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.626191 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.626171    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548ab8487a64671c8f29d89cabd4d700-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"548ab8487a64671c8f29d89cabd4d700\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.626278 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.626206    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.626278 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.626231    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.655986 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.655970    2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:38:55.656059 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.656007    2585 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.656059 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.656017    2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.680789 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.680774    2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:38:55.680848 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:55.680808    2585 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.726557 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.726537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548ab8487a64671c8f29d89cabd4d700-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"548ab8487a64671c8f29d89cabd4d700\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.726624 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.726562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.726624 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.726578    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.726713 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.726627    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.726713 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.726649    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.726713 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.726651    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548ab8487a64671c8f29d89cabd4d700-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"548ab8487a64671c8f29d89cabd4d700\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.868788 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.868718    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:55.874035 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:55.873819    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:38:57.565295 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.565222    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 19:38:57.565812 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.565400    2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:38:57.565812 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.565409    2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:38:57.565812 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.565413    2585 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:38:57.610538 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.610510    2585 apiserver.go:52] "Watching apiserver"
Nov 03 19:38:57.620870 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.620847    2585 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:38:57.621106 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.621089    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal","openshift-multus/multus-dp7w8","openshift-multus/multus-additional-cni-plugins-xb2qp","openshift-multus/network-metrics-daemon-bxkbj","kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal","openshift-image-registry/node-ca-km679","openshift-network-diagnostics/network-check-target-79zbh","openshift-cluster-node-tuning-operator/tuned-t75cv","openshift-dns/node-resolver-cjqb6","openshift-network-operator/iptables-alerter-cncml","openshift-ovn-kubernetes/ovnkube-node-w5md5","kube-system/global-pull-secret-syncer-2g52t","kube-system/konnectivity-agent-dlgxk"]
Nov 03 19:38:57.623614 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.623595    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.625690 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.625669    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.626885 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.626867    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:38:57.626885 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.626844    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:33:56 +0000 UTC" deadline="2028-04-24 13:56:49.180140732 +0000 UTC"
Nov 03 19:38:57.626987 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.626890    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12906h17m51.553252688s"
Nov 03 19:38:57.627029 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.627020    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:38:57.627129 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.627116    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:38:57.627252 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.627236    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xvgk\""
Nov 03 19:38:57.627867 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.627856    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.628423 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.628407    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:38:57.628602 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.628588    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:38:57.628661 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.628616    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rlc2f\""
Nov 03 19:38:57.628661 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.628624    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:38:57.628661 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.628653    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:38:57.630025 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.630009    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:57.630107 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.630088    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:38:57.630409 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.630393    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq9xl\""
Nov 03 19:38:57.630498 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.630444    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:38:57.630498 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.630457    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:38:57.632264 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.632251    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.634284 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.634272    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:38:57.634330 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.634319    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:38:57.636485 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.636469    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.636578 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.636499    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2rb8\""
Nov 03 19:38:57.638233 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.638219    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 19:38:57.641220 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.638639    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.641220 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.640818    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.643263 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.643248    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.645361 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645345    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.645457 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.645457 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.645398    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:38:57.645457 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645414    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6pl\" (UniqueName: \"kubernetes.io/projected/ce3c36fc-764d-4dc1-a515-000a6407542a-kube-api-access-zt6pl\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.645457 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645431    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.645457 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645454    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbknc\" (UniqueName: \"kubernetes.io/projected/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kube-api-access-qbknc\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.645697 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645514    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qspcq\" (UniqueName: \"kubernetes.io/projected/7b370ef7-e181-4b13-ba85-48090536f25c-kube-api-access-qspcq\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.645697 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645589    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-registration-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.645697 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645646    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-multus\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.645865 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645687    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzg8t\" (UniqueName: \"kubernetes.io/projected/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-kube-api-access-mzg8t\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.645865 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645721    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.645865 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysconfig\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.645865 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.645865 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.645819    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-multus-certs\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646354 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646335    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-etc-tuned\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.646432 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646379    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3c36fc-764d-4dc1-a515-000a6407542a-hosts-file\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.646432 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646400    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-modprobe-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.646503 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646450    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-socket-dir-parent\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646503 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646490    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-netns\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646573 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.646573 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-host\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.646573 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646553    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-multus-daemon-config\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646662 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646575    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbd5\" (UniqueName: \"kubernetes.io/projected/e5ef855a-3c98-47fd-8559-bb40ad4db198-kube-api-access-6zbd5\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.646662 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646594    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-sys-fs\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.646662 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646628    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-cni-binary-copy\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646662 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-conf-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646798 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646667    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-device-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.646798 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646685    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-cnibin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646798 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646731    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbx9\" (UniqueName: \"kubernetes.io/projected/26f64000-db36-453a-a425-a19b06a36f30-kube-api-access-vkbx9\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646798 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646753    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.646798 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646772    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.646798 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646789    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-kubelet\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.646975 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646807    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:57.646975 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646828    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ef855a-3c98-47fd-8559-bb40ad4db198-serviceca\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.646975 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646848    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c2e6638-f5a0-4258-a866-43417ff0f0c4-iptables-alerter-script\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.646975 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.646962    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-systemd\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647137 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647122    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-run\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647174 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-tmp\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647174 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647167    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-lib-modules\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647248 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647220    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-kubernetes\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647285 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647252    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-conf\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647456 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647377    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-k8s-cni-cncf-io\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.647456 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647423    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4p94\" (UniqueName: \"kubernetes.io/projected/2c2e6638-f5a0-4258-a866-43417ff0f0c4-kube-api-access-d4p94\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.647580 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647472    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-socket-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.647580 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-sys\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647580 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647535    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c36fc-764d-4dc1-a515-000a6407542a-tmp-dir\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.647769 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647593    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-var-lib-kubelet\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.647769 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647664    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-hostroot\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.647769 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647698    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.647769 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647732    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnkn\" (UniqueName: \"kubernetes.io/projected/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-kube-api-access-ctnkn\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:57.647769 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647757    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ef855a-3c98-47fd-8559-bb40ad4db198-host\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647777    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647789    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647821    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-system-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647851    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-bin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-etc-kubernetes\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.647979    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cnibin\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.648015    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.648079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.648046    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c2e6638-f5a0-4258-a866-43417ff0f0c4-host-slash\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.652384 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.652366    2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:38:57.667514 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.667493    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pp4wz"
Nov 03 19:38:57.675260 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.675244    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pp4wz"
Nov 03 19:38:57.727662 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727637    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:38:57.727782 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727680    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:38:57.727782 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727691    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:38:57.727782 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727697    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:38:57.727782 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727750    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:38:57.727782 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727757    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:38:57.727972 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727796    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:38:57.727972 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727881    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:38:57.727972 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.727894    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:38:57.728800 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.728782    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:38:57.728899 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.728831    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:38:57.728899 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.728849    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:38:57.728899 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.728859    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:38:57.729017 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.728917    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:38:57.729050 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.729042    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:38:57.729125 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.729112    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:38:57.729170 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.729138    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d4wgq\""
Nov 03 19:38:57.729247 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.729224    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzp2g\""
Nov 03 19:38:57.730370 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.730356    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbmvx\""
Nov 03 19:38:57.730370 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.730362    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxqwg\""
Nov 03 19:38:57.730456 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.730445    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:38:57.730489 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.730450    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:38:57.730489 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.730464    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmwsk\""
Nov 03 19:38:57.740716 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.740697    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:38:57.748712 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748692    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-conf\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.748851 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748728    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-kubelet\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.748851 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748752    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-systemd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.748851 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748809    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-k8s-cni-cncf-io\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.748971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748844    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4p94\" (UniqueName: \"kubernetes.io/projected/2c2e6638-f5a0-4258-a866-43417ff0f0c4-kube-api-access-d4p94\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.748971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748857    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-conf\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.748971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748894    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-socket-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.748971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748920    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-sys\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.748971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748940    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-ovn\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.748971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748954    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c36fc-764d-4dc1-a515-000a6407542a-tmp-dir\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.748971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748969    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-k8s-cni-cncf-io\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.748975    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-var-lib-kubelet\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749008    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-sys\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749054    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovn-node-metrics-cert\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749066    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-var-lib-kubelet\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749056    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-socket-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-hostroot\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnkn\" (UniqueName: \"kubernetes.io/projected/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-kube-api-access-ctnkn\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749150    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ef855a-3c98-47fd-8559-bb40ad4db198-host\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749168    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-hostroot\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749186    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-node-log\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749218    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ef855a-3c98-47fd-8559-bb40ad4db198-host\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-env-overrides\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749267    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjr9h\" (UniqueName: \"kubernetes.io/projected/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-kube-api-access-qjr9h\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749284    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-system-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.749314 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749298    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-bin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749312    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-etc-kubernetes\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749312    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c36fc-764d-4dc1-a515-000a6407542a-tmp-dir\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cnibin\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749348    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749356    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-system-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749362    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c2e6638-f5a0-4258-a866-43417ff0f0c4-host-slash\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749388    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c2e6638-f5a0-4258-a866-43417ff0f0c4-host-slash\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749389    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-etc-kubernetes\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749412    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-etc-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749418    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cnibin\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749423    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-bin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749428    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749495    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6pl\" (UniqueName: \"kubernetes.io/projected/ce3c36fc-764d-4dc1-a515-000a6407542a-kube-api-access-zt6pl\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749517    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749556    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbknc\" (UniqueName: \"kubernetes.io/projected/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kube-api-access-qbknc\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.750140 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749622    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspcq\" (UniqueName: \"kubernetes.io/projected/7b370ef7-e181-4b13-ba85-48090536f25c-kube-api-access-qspcq\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749689    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749779    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749832    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc369cd-d32a-4cad-9da6-d5bfd899591a-agent-certs\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749872    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749879    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc369cd-d32a-4cad-9da6-d5bfd899591a-konnectivity-ca\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749911    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-registration-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749937    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-netns\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749961    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-log-socket\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749969    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-registration-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.749976    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750002    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-kubelet-config\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750027    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-dbus\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750047    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-multus\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750076    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzg8t\" (UniqueName: \"kubernetes.io/projected/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-kube-api-access-mzg8t\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.750951 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750093    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750149    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysconfig\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750231    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-multus\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750243    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-var-lib-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750260    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysconfig\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750268    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-config\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750289    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-multus-certs\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-etc-tuned\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750395    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-systemd-units\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750429    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3c36fc-764d-4dc1-a515-000a6407542a-hosts-file\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750432    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750400    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-multus-certs\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-modprobe-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-slash\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-netd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.751645 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750549    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-socket-dir-parent\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750588    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-modprobe-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750600    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3c36fc-764d-4dc1-a515-000a6407542a-hosts-file\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750612    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-netns\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750634    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-socket-dir-parent\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750651    2585 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750668    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-host\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750717    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-bin\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750724    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-netns\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750727    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-host\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750744    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-multus-daemon-config\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750766    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbd5\" (UniqueName: \"kubernetes.io/projected/e5ef855a-3c98-47fd-8559-bb40ad4db198-kube-api-access-6zbd5\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750802    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-sys-fs\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750867    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-sys-fs\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-cni-binary-copy\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.750965    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-conf-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752341 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-device-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751002    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-conf-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751033    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751061    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-cnibin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751069    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-device-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751086    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbx9\" (UniqueName: \"kubernetes.io/projected/26f64000-db36-453a-a425-a19b06a36f30-kube-api-access-vkbx9\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751132    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751146    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-cnibin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751238    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-script-lib\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751288    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-kubelet\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751345    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751374    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ef855a-3c98-47fd-8559-bb40ad4db198-serviceca\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751379    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-kubelet\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751412    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c2e6638-f5a0-4258-a866-43417ff0f0c4-iptables-alerter-script\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751435    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-systemd\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.751451    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:38:57.752918 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751456    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-run\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-tmp\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751506    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-lib-modules\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.751528    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs podName:b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:58.251494417 +0000 UTC m=+2.001595009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs") pod "network-metrics-daemon-bxkbj" (UID: "b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751557    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-kubernetes\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751594    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-lib-modules\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751613    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-kubernetes\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751653    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-systemd\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751707    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.751739    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-run\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.752052    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.752054    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-multus-daemon-config\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.752221    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-cni-binary-copy\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.752429    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ef855a-3c98-47fd-8559-bb40ad4db198-serviceca\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.753451 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.752472    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c2e6638-f5a0-4258-a866-43417ff0f0c4-iptables-alerter-script\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.754011 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.753989    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-etc-tuned\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.754079 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.754037    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-tmp\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.755756 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.755695    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:38:57.755890 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.755768    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:38:57.755890 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.755781    2585 projected.go:196] Error preparing data for projected volume kube-api-access-tzkk5 for pod openshift-network-diagnostics/network-check-target-79zbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:38:57.755890 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.755833    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5 podName:00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee nodeName:}" failed. No retries permitted until 2026-11-03 19:38:58.255817574 +0000 UTC m=+2.005918159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tzkk5" (UniqueName: "kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5") pod "network-check-target-79zbh" (UID: "00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:38:57.758146 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.758123    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6pl\" (UniqueName: \"kubernetes.io/projected/ce3c36fc-764d-4dc1-a515-000a6407542a-kube-api-access-zt6pl\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:57.758288 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.758273    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzg8t\" (UniqueName: \"kubernetes.io/projected/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-kube-api-access-mzg8t\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.758456 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.758434    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4p94\" (UniqueName: \"kubernetes.io/projected/2c2e6638-f5a0-4258-a866-43417ff0f0c4-kube-api-access-d4p94\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:57.758999 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.758977    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnkn\" (UniqueName: \"kubernetes.io/projected/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-kube-api-access-ctnkn\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:57.759218 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.759188    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbknc\" (UniqueName: \"kubernetes.io/projected/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kube-api-access-qbknc\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.759632 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.759607    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbx9\" (UniqueName: \"kubernetes.io/projected/26f64000-db36-453a-a425-a19b06a36f30-kube-api-access-vkbx9\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.759851 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.759832    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspcq\" (UniqueName: \"kubernetes.io/projected/7b370ef7-e181-4b13-ba85-48090536f25c-kube-api-access-qspcq\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:57.760569 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.760550    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbd5\" (UniqueName: \"kubernetes.io/projected/e5ef855a-3c98-47fd-8559-bb40ad4db198-kube-api-access-6zbd5\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.852117 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852038    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.852117 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852082    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852117 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852098    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-script-lib\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-kubelet\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852151    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-systemd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.852161    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-ovn\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:57.852249    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret podName:6b3aec1c-dcd1-4b89-86cc-898550befdad nodeName:}" failed. No retries permitted until 2026-11-03 19:38:58.352230537 +0000 UTC m=+2.102331132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret") pod "global-pull-secret-syncer-2g52t" (UID: "6b3aec1c-dcd1-4b89-86cc-898550befdad") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852246    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852260    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-systemd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852256    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-ovn\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852300    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-kubelet\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852394    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovn-node-metrics-cert\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852448 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-node-log\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852468    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-env-overrides\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852489    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-node-log\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852491    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjr9h\" (UniqueName: \"kubernetes.io/projected/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-kube-api-access-qjr9h\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-etc-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852552    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852590    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc369cd-d32a-4cad-9da6-d5bfd899591a-agent-certs\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852613    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc369cd-d32a-4cad-9da6-d5bfd899591a-konnectivity-ca\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852635    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852636    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852673    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-netns\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852638    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-netns\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852687    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-etc-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852702    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-log-socket\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852733    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-kubelet-config\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852746    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-log-socket\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.852905 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852754    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-dbus\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852791    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-var-lib-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852793    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-kubelet-config\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852806    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-config\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852823    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-systemd-units\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852825    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-dbus\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852838    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-var-lib-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-slash\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852868    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-systemd-units\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852876    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-netd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852904    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-bin\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852944    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-bin\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852968    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-slash\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.852969    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-netd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853894 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.853440    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-script-lib\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853894 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.853596    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-env-overrides\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.853894 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.853777    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc369cd-d32a-4cad-9da6-d5bfd899591a-konnectivity-ca\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:57.853894 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.853780    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-config\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.854805 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.854784    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovn-node-metrics-cert\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.855416 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.855397    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc369cd-d32a-4cad-9da6-d5bfd899591a-agent-certs\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:57.862266 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.862248    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjr9h\" (UniqueName: \"kubernetes.io/projected/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-kube-api-access-qjr9h\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:57.886122 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:57.886099    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548ab8487a64671c8f29d89cabd4d700.slice/crio-179fe18ca01b9400756c7ca2d092ed51c0fb86ed9b338c9790d9f0abac7a8d96 WatchSource:0}: Error finding container 179fe18ca01b9400756c7ca2d092ed51c0fb86ed9b338c9790d9f0abac7a8d96: Status 404 returned error can't find the container with id 179fe18ca01b9400756c7ca2d092ed51c0fb86ed9b338c9790d9f0abac7a8d96
Nov 03 19:38:57.894061 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.894043    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:38:57.895731 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:57.895711    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716ea44535b80bd6ca43cdb7c07af7ab.slice/crio-07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66 WatchSource:0}: Error finding container 07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66: Status 404 returned error can't find the container with id 07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66
Nov 03 19:38:57.953019 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.952993    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:38:57.962715 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:57.962694    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc78c3f5_3003_468d_82b1_bf29d59e5ea2.slice/crio-8aadc46d634086940923128e7ab65c7eeeb939f7477497be0653d4fa63908cbe WatchSource:0}: Error finding container 8aadc46d634086940923128e7ab65c7eeeb939f7477497be0653d4fa63908cbe: Status 404 returned error can't find the container with id 8aadc46d634086940923128e7ab65c7eeeb939f7477497be0653d4fa63908cbe
Nov 03 19:38:57.963911 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.963891    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dp7w8"
Nov 03 19:38:57.971619 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:57.971599    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f64000_db36_453a_a425_a19b06a36f30.slice/crio-1d7dd793fe1b15f60cfb0a39f1e8005c883f8cd7bed5a0fefb2294f779796263 WatchSource:0}: Error finding container 1d7dd793fe1b15f60cfb0a39f1e8005c883f8cd7bed5a0fefb2294f779796263: Status 404 returned error can't find the container with id 1d7dd793fe1b15f60cfb0a39f1e8005c883f8cd7bed5a0fefb2294f779796263
Nov 03 19:38:57.982514 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.982497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:38:57.985025 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:57.985012    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-km679"
Nov 03 19:38:57.990556 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:57.990534    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbc3cd0_c3c4_4355_aca3_fa215c9c8dc2.slice/crio-1446892fef638100aee1a3f8e99093270174937a14745471cab1a56244fed52c WatchSource:0}: Error finding container 1446892fef638100aee1a3f8e99093270174937a14745471cab1a56244fed52c: Status 404 returned error can't find the container with id 1446892fef638100aee1a3f8e99093270174937a14745471cab1a56244fed52c
Nov 03 19:38:57.993461 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:57.993441    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ef855a_3c98_47fd_8559_bb40ad4db198.slice/crio-4b00fb0b96fbf679b568c28aa798c4cd9c8113cfe144e0fcf1bdbd414bccab3a WatchSource:0}: Error finding container 4b00fb0b96fbf679b568c28aa798c4cd9c8113cfe144e0fcf1bdbd414bccab3a: Status 404 returned error can't find the container with id 4b00fb0b96fbf679b568c28aa798c4cd9c8113cfe144e0fcf1bdbd414bccab3a
Nov 03 19:38:58.003086 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.003071    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:38:58.011238 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:58.011217    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b370ef7_e181_4b13_ba85_48090536f25c.slice/crio-99c1dd5d66a1f4f6df28213bf8e260af1b05fcffc1569b4d34c64e07659e41aa WatchSource:0}: Error finding container 99c1dd5d66a1f4f6df28213bf8e260af1b05fcffc1569b4d34c64e07659e41aa: Status 404 returned error can't find the container with id 99c1dd5d66a1f4f6df28213bf8e260af1b05fcffc1569b4d34c64e07659e41aa
Nov 03 19:38:58.020913 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.020896    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:38:58.029048 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:58.029030    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3c36fc_764d_4dc1_a515_000a6407542a.slice/crio-a65d7756f3ce5da24700580ef223a05adc701675a7463fa5fba1a9bccf530ce3 WatchSource:0}: Error finding container a65d7756f3ce5da24700580ef223a05adc701675a7463fa5fba1a9bccf530ce3: Status 404 returned error can't find the container with id a65d7756f3ce5da24700580ef223a05adc701675a7463fa5fba1a9bccf530ce3
Nov 03 19:38:58.045092 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.045073    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:38:58.053540 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:58.053522    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e6638_f5a0_4258_a866_43417ff0f0c4.slice/crio-82b208109713187177ff5ad0ff5b90d3247f5592764737a15dae8db3c39950eb WatchSource:0}: Error finding container 82b208109713187177ff5ad0ff5b90d3247f5592764737a15dae8db3c39950eb: Status 404 returned error can't find the container with id 82b208109713187177ff5ad0ff5b90d3247f5592764737a15dae8db3c39950eb
Nov 03 19:38:58.068593 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.068572    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:38:58.075655 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.075640    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:38:58.077139 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:58.077120    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec13f7e_eee1_4082_bc4d_d71e59ef78c1.slice/crio-632b44e40acc360cc62dd8eb6572293dd105ce73443bca8909ffbb2ab0cb0dd0 WatchSource:0}: Error finding container 632b44e40acc360cc62dd8eb6572293dd105ce73443bca8909ffbb2ab0cb0dd0: Status 404 returned error can't find the container with id 632b44e40acc360cc62dd8eb6572293dd105ce73443bca8909ffbb2ab0cb0dd0
Nov 03 19:38:58.083434 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:38:58.083415    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc369cd_d32a_4cad_9da6_d5bfd899591a.slice/crio-863d28a47b9f6679bf41d3df660c67eba3b824c17d92effdc5159e7e738d2ea2 WatchSource:0}: Error finding container 863d28a47b9f6679bf41d3df660c67eba3b824c17d92effdc5159e7e738d2ea2: Status 404 returned error can't find the container with id 863d28a47b9f6679bf41d3df660c67eba3b824c17d92effdc5159e7e738d2ea2
Nov 03 19:38:58.085580 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.085559    2585 kuberuntime_manager.go:1707] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-dlgxk_kube-system(0fc369cd-d32a-4cad-9da6-d5bfd899591a): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Nov 03 19:38:58.085646 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.085592    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-dlgxk" podUID="0fc369cd-d32a-4cad-9da6-d5bfd899591a"
Nov 03 19:38:58.255261 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.255173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:58.255413 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.255337    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:38:58.255413 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.255407    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs podName:b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41 nodeName:}" failed. No retries permitted until 2026-11-03 19:38:59.255389484 +0000 UTC m=+3.005490073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs") pod "network-metrics-daemon-bxkbj" (UID: "b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:38:58.356173 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.356133    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:58.356356 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.356238    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:38:58.356356 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.356295    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:38:58.356465 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.356361    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:38:58.356465 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.356369    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret podName:6b3aec1c-dcd1-4b89-86cc-898550befdad nodeName:}" failed. No retries permitted until 2026-11-03 19:38:59.356351135 +0000 UTC m=+3.106451726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret") pod "global-pull-secret-syncer-2g52t" (UID: "6b3aec1c-dcd1-4b89-86cc-898550befdad") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:38:58.356465 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.356383    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:38:58.356465 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.356395    2585 projected.go:196] Error preparing data for projected volume kube-api-access-tzkk5 for pod openshift-network-diagnostics/network-check-target-79zbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:38:58.356465 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.356442    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5 podName:00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee nodeName:}" failed. No retries permitted until 2026-11-03 19:38:59.356426397 +0000 UTC m=+3.106526986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzkk5" (UniqueName: "kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5") pod "network-check-target-79zbh" (UID: "00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:38:58.677690 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.677551    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:33:57 +0000 UTC" deadline="2028-06-03 11:39:28.138309895 +0000 UTC"
Nov 03 19:38:58.677690 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.677619    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13864h0m29.460694992s"
Nov 03 19:38:58.734583 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.734554    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:58.734778 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.734686    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:38:58.757549 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.757458    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dlgxk" event={"ID":"0fc369cd-d32a-4cad-9da6-d5bfd899591a","Type":"ContainerStarted","Data":"863d28a47b9f6679bf41d3df660c67eba3b824c17d92effdc5159e7e738d2ea2"}
Nov 03 19:38:58.762910 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:58.762876    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef3d8a5c5a290fbfb7b3688e9a224e9c47d9f3326f05a0f73577656437e3f406\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-dlgxk" podUID="0fc369cd-d32a-4cad-9da6-d5bfd899591a"
Nov 03 19:38:58.763425 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.763378    2585 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:38:58.768456 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.768420    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cncml" event={"ID":"2c2e6638-f5a0-4258-a866-43417ff0f0c4","Type":"ContainerStarted","Data":"82b208109713187177ff5ad0ff5b90d3247f5592764737a15dae8db3c39950eb"}
Nov 03 19:38:58.791680 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.791648    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjqb6" event={"ID":"ce3c36fc-764d-4dc1-a515-000a6407542a","Type":"ContainerStarted","Data":"a65d7756f3ce5da24700580ef223a05adc701675a7463fa5fba1a9bccf530ce3"}
Nov 03 19:38:58.796836 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.796808    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km679" event={"ID":"e5ef855a-3c98-47fd-8559-bb40ad4db198","Type":"ContainerStarted","Data":"4b00fb0b96fbf679b568c28aa798c4cd9c8113cfe144e0fcf1bdbd414bccab3a"}
Nov 03 19:38:58.812850 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.812822    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dp7w8" event={"ID":"26f64000-db36-453a-a425-a19b06a36f30","Type":"ContainerStarted","Data":"1d7dd793fe1b15f60cfb0a39f1e8005c883f8cd7bed5a0fefb2294f779796263"}
Nov 03 19:38:58.821119 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.821086    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"632b44e40acc360cc62dd8eb6572293dd105ce73443bca8909ffbb2ab0cb0dd0"}
Nov 03 19:38:58.831391 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.831299    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t75cv" event={"ID":"7b370ef7-e181-4b13-ba85-48090536f25c","Type":"ContainerStarted","Data":"99c1dd5d66a1f4f6df28213bf8e260af1b05fcffc1569b4d34c64e07659e41aa"}
Nov 03 19:38:58.841764 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.841732    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerStarted","Data":"1446892fef638100aee1a3f8e99093270174937a14745471cab1a56244fed52c"}
Nov 03 19:38:58.853833 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.853804    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t" event={"ID":"bc78c3f5-3003-468d-82b1-bf29d59e5ea2","Type":"ContainerStarted","Data":"8aadc46d634086940923128e7ab65c7eeeb939f7477497be0653d4fa63908cbe"}
Nov 03 19:38:58.857283 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.857259    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"716ea44535b80bd6ca43cdb7c07af7ab","Type":"ContainerStarted","Data":"07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66"}
Nov 03 19:38:58.860601 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.860576    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"548ab8487a64671c8f29d89cabd4d700","Type":"ContainerStarted","Data":"179fe18ca01b9400756c7ca2d092ed51c0fb86ed9b338c9790d9f0abac7a8d96"}
Nov 03 19:38:58.881032 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.880908    2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:38:58.960907 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:58.960823    2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:38:59.265340 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:59.264673    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:38:59.265340 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.264815    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:38:59.265340 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.264874    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs podName:b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:01.264857393 +0000 UTC m=+5.014957979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs") pod "network-metrics-daemon-bxkbj" (UID: "b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:38:59.365886 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:59.365849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:59.366087 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:59.365912    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:38:59.366087 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.366032    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:38:59.366087 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.366045    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:38:59.366087 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.366054    2585 projected.go:196] Error preparing data for projected volume kube-api-access-tzkk5 for pod openshift-network-diagnostics/network-check-target-79zbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:38:59.366325 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.366093    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5 podName:00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee nodeName:}" failed. No retries permitted until 2026-11-03 19:39:01.366081388 +0000 UTC m=+5.116181963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzkk5" (UniqueName: "kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5") pod "network-check-target-79zbh" (UID: "00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:38:59.366450 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.366436    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:38:59.366503 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.366477    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret podName:6b3aec1c-dcd1-4b89-86cc-898550befdad nodeName:}" failed. No retries permitted until 2026-11-03 19:39:01.366465939 +0000 UTC m=+5.116566511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret") pod "global-pull-secret-syncer-2g52t" (UID: "6b3aec1c-dcd1-4b89-86cc-898550befdad") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:38:59.678168 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:59.678080    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:33:57 +0000 UTC" deadline="2028-03-29 01:19:14.126531005 +0000 UTC"
Nov 03 19:38:59.678168 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:59.678114    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12269h40m14.448420825s"
Nov 03 19:38:59.732626 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:59.732594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:38:59.732796 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.732754    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:38:59.733250 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:38:59.733230    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:38:59.733349 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:38:59.733328    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:00.735775 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:00.735740    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:00.736273 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:00.735884    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:01.285711 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.285262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:01.285711 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.285377    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:01.285711 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.285426    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs podName:b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:05.285413464 +0000 UTC m=+9.035514035 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs") pod "network-metrics-daemon-bxkbj" (UID: "b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:01.387021 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.386162    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:01.387021 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.386255    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:01.387021 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.386548    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:01.387021 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.386625    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret podName:6b3aec1c-dcd1-4b89-86cc-898550befdad nodeName:}" failed. No retries permitted until 2026-11-03 19:39:05.386607184 +0000 UTC m=+9.136707768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret") pod "global-pull-secret-syncer-2g52t" (UID: "6b3aec1c-dcd1-4b89-86cc-898550befdad") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:01.387628 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.387521    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:01.387628 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.387548    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:01.387628 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.387561    2585 projected.go:196] Error preparing data for projected volume kube-api-access-tzkk5 for pod openshift-network-diagnostics/network-check-target-79zbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:01.388110 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.388014    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5 podName:00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee nodeName:}" failed. No retries permitted until 2026-11-03 19:39:05.387995152 +0000 UTC m=+9.138095740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzkk5" (UniqueName: "kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5") pod "network-check-target-79zbh" (UID: "00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:01.732732 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.732540    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:01.732951 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.732814    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:01.733138 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.733123    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:01.733248 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:01.733220    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:01.878227 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.878109    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="27a7b090c3c93724f6432077c078320d8a74bc531fd61fbb91daab65cb5f0eaf" exitCode=0
Nov 03 19:39:01.878227 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.878190    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerDied","Data":"27a7b090c3c93724f6432077c078320d8a74bc531fd61fbb91daab65cb5f0eaf"}
Nov 03 19:39:01.884875 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.884848    2585 generic.go:356] "Generic (PLEG): container finished" podID="716ea44535b80bd6ca43cdb7c07af7ab" containerID="21741986a3156f047d0cb7e458a14048acd374927ca02526a13b5640c42aa1b1" exitCode=0
Nov 03 19:39:01.884981 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:01.884886    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"716ea44535b80bd6ca43cdb7c07af7ab","Type":"ContainerDied","Data":"21741986a3156f047d0cb7e458a14048acd374927ca02526a13b5640c42aa1b1"}
Nov 03 19:39:02.735154 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:02.735109    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:02.735320 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:02.735286    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:02.890713 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:02.890677    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"716ea44535b80bd6ca43cdb7c07af7ab","Type":"ContainerStarted","Data":"d605ffb34fdc1288607284d50c2a48ba03b2ca6dc773fbefe64a9fe6d039d427"}
Nov 03 19:39:02.912530 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:02.912462    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"548ab8487a64671c8f29d89cabd4d700","Type":"ContainerStarted","Data":"a25696f666694efab7aa7bffea43068b6a96b275033b2fe40e60870e199b83c1"}
Nov 03 19:39:02.921465 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:02.921423    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal" podStartSLOduration=6.921402112 podStartE2EDuration="6.921402112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:02.920975729 +0000 UTC m=+6.671076319" watchObservedRunningTime="2026-11-03 19:39:02.921402112 +0000 UTC m=+6.671502706"
Nov 03 19:39:03.733057 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:03.732989    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:03.733230 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:03.733100    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:03.733315 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:03.733282    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:03.733400 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:03.733370    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:03.921098 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:03.921061    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-km679" event={"ID":"e5ef855a-3c98-47fd-8559-bb40ad4db198","Type":"ContainerStarted","Data":"8e8a4aca879d940366393f19e3f55b714184642378560fa0bf17da243e7a77fc"}
Nov 03 19:39:03.935512 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:03.935458    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal" podStartSLOduration=7.935442014 podStartE2EDuration="7.935442014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:02.937535766 +0000 UTC m=+6.687636363" watchObservedRunningTime="2026-11-03 19:39:03.935442014 +0000 UTC m=+7.685542609"
Nov 03 19:39:04.734438 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:04.734408    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:04.734610 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:04.734518    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:04.926348 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:04.926262    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t" event={"ID":"bc78c3f5-3003-468d-82b1-bf29d59e5ea2","Type":"ContainerStarted","Data":"584966e2ebc60124c0a140c5841e9c49aaec45b3803cb8edf06fb8f508ce208a"}
Nov 03 19:39:05.333641 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:05.333045    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:05.333641 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.333280    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:05.333641 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.333390    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs podName:b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:13.333346514 +0000 UTC m=+17.083447088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs") pod "network-metrics-daemon-bxkbj" (UID: "b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:05.433809    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:05.433881    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.434047    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.434070    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.434084    2585 projected.go:196] Error preparing data for projected volume kube-api-access-tzkk5 for pod openshift-network-diagnostics/network-check-target-79zbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.434217    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5 podName:00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee nodeName:}" failed. No retries permitted until 2026-11-03 19:39:13.434169262 +0000 UTC m=+17.184269848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzkk5" (UniqueName: "kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5") pod "network-check-target-79zbh" (UID: "00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.434614    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:05.434707 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.434668    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret podName:6b3aec1c-dcd1-4b89-86cc-898550befdad nodeName:}" failed. No retries permitted until 2026-11-03 19:39:13.434652877 +0000 UTC m=+17.184753463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret") pod "global-pull-secret-syncer-2g52t" (UID: "6b3aec1c-dcd1-4b89-86cc-898550befdad") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:05.732345 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:05.732308    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:05.732499 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.732480    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:05.732967 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:05.732947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:05.733105 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:05.733072    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:05.817652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:05.817397    2585 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:39:05.930430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:05.930344    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t" event={"ID":"bc78c3f5-3003-468d-82b1-bf29d59e5ea2","Type":"ContainerStarted","Data":"57507b5954ae2448d48fd7d2c196c4646a7bfec7384268db54897b4c201a63e7"}
Nov 03 19:39:06.737482 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.737447    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:06.737644 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:06.737570    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:06.738565 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.738472    2585 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:39:05.817419909Z","UUID":"fb1bb163-f91d-4a6d-99bd-d6d5e5311b9b","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:39:06.740090 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.740068    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:39:06.740223 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.740099    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:39:06.935855 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.935231    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjqb6" event={"ID":"ce3c36fc-764d-4dc1-a515-000a6407542a","Type":"ContainerStarted","Data":"58ae649f934699765a502b09fd951d7dfdbc80a5f246ac7cdacac7c6c4eaca0c"}
Nov 03 19:39:06.938290 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.938267    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dp7w8" event={"ID":"26f64000-db36-453a-a425-a19b06a36f30","Type":"ContainerStarted","Data":"167d6bcc091ef05a71ce58f44304041002dd8a2e887eaa00fd3b083ad9a8d2d8"}
Nov 03 19:39:06.940726 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.940697    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="1a37025da157ac12d55eb37facd740b509af8f5484f75b8bcd85c44d5aacd4c8" exitCode=0
Nov 03 19:39:06.940806 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.940734    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerDied","Data":"1a37025da157ac12d55eb37facd740b509af8f5484f75b8bcd85c44d5aacd4c8"}
Nov 03 19:39:06.950404 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.950357    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-km679" podStartSLOduration=5.675650331 podStartE2EDuration="10.950343313s" totalImagesPullingTime="5.274692982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:03.936469536 +0000 UTC m=+7.686570136" watchObservedRunningTime="2026-11-03 19:39:06.950343313 +0000 UTC m=+10.700443931"
Nov 03 19:39:06.950633 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.950602    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cjqb6" podStartSLOduration=2.712914699 podStartE2EDuration="10.9505938s" totalImagesPullingTime="8.237679101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:06.950326998 +0000 UTC m=+10.700427592" watchObservedRunningTime="2026-11-03 19:39:06.9505938 +0000 UTC m=+10.700694396"
Nov 03 19:39:06.989761 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:06.989713    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dp7w8" podStartSLOduration=3.098977482 podStartE2EDuration="10.989696415s" totalImagesPullingTime="7.890718933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:06.988789659 +0000 UTC m=+10.738890253" watchObservedRunningTime="2026-11-03 19:39:06.989696415 +0000 UTC m=+10.739797011"
Nov 03 19:39:07.732446 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.732253    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:07.732599 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.732313    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:07.732599 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:07.732529    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:07.732715 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:07.732659    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:07.945115 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.945026    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t" event={"ID":"bc78c3f5-3003-468d-82b1-bf29d59e5ea2","Type":"ContainerStarted","Data":"5449dd970c450670d0918ee409f18bb93770c520aadea6bffe95336fd4372873"}
Nov 03 19:39:07.947136 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.947108    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cncml" event={"ID":"2c2e6638-f5a0-4258-a866-43417ff0f0c4","Type":"ContainerStarted","Data":"45fed211ffb160959bd5dcf08b6e51ddaa09e29e8832a2e455da82e21a723dce"}
Nov 03 19:39:07.951806 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.951780    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="04e250a911388ad4a40dafc66fae03281b6bd332aa8ce87417d9f6ce6ba28e46" exitCode=0
Nov 03 19:39:07.951931 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.951877    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerDied","Data":"04e250a911388ad4a40dafc66fae03281b6bd332aa8ce87417d9f6ce6ba28e46"}
Nov 03 19:39:07.963714 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.963669    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t" podStartSLOduration=3.408194968 podStartE2EDuration="11.963654059s" totalImagesPullingTime="8.555459091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:07.963148749 +0000 UTC m=+11.713249343" watchObservedRunningTime="2026-11-03 19:39:07.963654059 +0000 UTC m=+11.713754656"
Nov 03 19:39:07.976869 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:07.976830    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cncml" podStartSLOduration=3.9995042830000003 podStartE2EDuration="11.976819556s" totalImagesPullingTime="7.977315273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:07.976567573 +0000 UTC m=+11.726668173" watchObservedRunningTime="2026-11-03 19:39:07.976819556 +0000 UTC m=+11.726920149"
Nov 03 19:39:08.756082 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:08.756050    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:08.756270 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:08.756217    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:08.955629 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:08.955157    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t75cv" event={"ID":"7b370ef7-e181-4b13-ba85-48090536f25c","Type":"ContainerStarted","Data":"8d0547f26565eb3c966815b190b983c1849fae687446f1250ebefc679630156a"}
Nov 03 19:39:08.957884 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:08.957854    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="e35995db7a430e9d68d8a826a7947256e2713d144fd788668ab86930318303a1" exitCode=0
Nov 03 19:39:08.958007 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:08.957920    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerDied","Data":"e35995db7a430e9d68d8a826a7947256e2713d144fd788668ab86930318303a1"}
Nov 03 19:39:08.971124 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:08.971076    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t75cv" podStartSLOduration=2.934394995 podStartE2EDuration="12.971061848s" totalImagesPullingTime="10.036666853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:08.970591139 +0000 UTC m=+12.720691735" watchObservedRunningTime="2026-11-03 19:39:08.971061848 +0000 UTC m=+12.721162439"
Nov 03 19:39:09.732871 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:09.732833    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:09.733040 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:09.732968    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:09.733324 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:09.733300    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:09.733433 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:09.733408    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:10.732759 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:10.732728    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:10.733425 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:10.732879    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:11.732652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:11.732428    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:11.732843 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:11.732437    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:11.732843 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:11.732746    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:11.732843 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:11.732806    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:11.967494 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:11.967457    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="c3c650924a31d9b8fc0bcda12197e533943972dce59b773d6c6175d741ae1f33" exitCode=0
Nov 03 19:39:11.967649 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:11.967526    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerDied","Data":"c3c650924a31d9b8fc0bcda12197e533943972dce59b773d6c6175d741ae1f33"}
Nov 03 19:39:12.732545 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:12.732504    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:12.732689 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:12.732635    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:12.974853 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:12.974812    2585 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="72b4d4fd4f8aebe8647c477f39952248265ef537a0826f13362b0c2d85443523" exitCode=0
Nov 03 19:39:12.974853 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:12.974873    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerDied","Data":"72b4d4fd4f8aebe8647c477f39952248265ef537a0826f13362b0c2d85443523"}
Nov 03 19:39:13.394483 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.394322    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:13.394592 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.394458    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:13.394592 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.394551    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs podName:b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41 nodeName:}" failed. No retries permitted until 2026-11-03 19:39:29.394537892 +0000 UTC m=+33.144638476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs") pod "network-metrics-daemon-bxkbj" (UID: "b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:13.495074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.495051    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:13.495178 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.495096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:13.495304 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.495285    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:13.495421 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.495309    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:13.495421 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.495322    2585 projected.go:196] Error preparing data for projected volume kube-api-access-tzkk5 for pod openshift-network-diagnostics/network-check-target-79zbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:13.495421 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.495370    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5 podName:00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee nodeName:}" failed. No retries permitted until 2026-11-03 19:39:29.495353084 +0000 UTC m=+33.245453675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzkk5" (UniqueName: "kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5") pod "network-check-target-79zbh" (UID: "00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:13.495421 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.495286    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:13.495421 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.495409    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret podName:6b3aec1c-dcd1-4b89-86cc-898550befdad nodeName:}" failed. No retries permitted until 2026-11-03 19:39:29.49540035 +0000 UTC m=+33.245500947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret") pod "global-pull-secret-syncer-2g52t" (UID: "6b3aec1c-dcd1-4b89-86cc-898550befdad") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:13.732624 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.732594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:13.732793 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.732594    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:13.732793 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.732713    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:13.732793 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:13.732757    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:13.979882 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.979848    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"edfa90b7080fcf46158d12ba71dda3b20426af8489793f5658ce8bc830f73498"}
Nov 03 19:39:13.980370 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.979933    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"47cf38b44406e540db51e9df1cb14c31bb6029c33aabacd5069474e4eda9824b"}
Nov 03 19:39:13.980370 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.979975    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"e69e9a7b5eca75dc4ae461fcf8df53e1f47d65388c6e4bb9e6ff51165509200f"}
Nov 03 19:39:13.980370 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.980014    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"f4cc6ddcad87a35828b94e9dd4d026f8a9e5890858f4897954083aa81990237b"}
Nov 03 19:39:13.980370 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.980043    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"458c85ddc9f3ac552682d8406c44121abd41f72ece0e525008a3c336909282d0"}
Nov 03 19:39:13.980370 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.980084    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"14ac2cb0c9d88f0fe3d9d71fd91ebc42f08ec9a87b2cede28a172a92ce78fab9"}
Nov 03 19:39:13.983691 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:13.983624    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" event={"ID":"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2","Type":"ContainerStarted","Data":"c80d9b6801eb53e91ab54b483fe2fc241297d16b989244bc5fc644b27881f725"}
Nov 03 19:39:14.004953 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:14.004906    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xb2qp" podStartSLOduration=7.3924925009999995 podStartE2EDuration="18.004893949s" totalImagesPullingTime="10.612401448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:14.00440605 +0000 UTC m=+17.754506656" watchObservedRunningTime="2026-11-03 19:39:14.004893949 +0000 UTC m=+17.754994571"
Nov 03 19:39:14.732090 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:14.732065    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:14.732287 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:14.732169    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:14.987243 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:14.987140    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dlgxk" event={"ID":"0fc369cd-d32a-4cad-9da6-d5bfd899591a","Type":"ContainerStarted","Data":"965e09635ce4944bf57cebd9588121dd47665bd588adacc2bfa265f8c85c0406"}
Nov 03 19:39:15.005665 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:15.005622    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dlgxk" podStartSLOduration=2.929682185 podStartE2EDuration="19.005610833s" totalImagesPullingTime="16.075928648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-11-03 19:39:15.003894332 +0000 UTC m=+18.753994925" watchObservedRunningTime="2026-11-03 19:39:15.005610833 +0000 UTC m=+18.755711426"
Nov 03 19:39:15.733041 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:15.733007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:15.733041 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:15.733034    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:15.733273 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:15.733108    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:15.733273 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:15.733230    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:15.991897 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:15.991862    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"a5bf50e7f0303de86bcbbb3a87e533ab3b1ecda33c9eb9586e897c4257fa332c"}
Nov 03 19:39:16.734160 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:16.734126    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:16.734338 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:16.734281    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:17.732266 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:17.732228    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:17.732266 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:17.732263    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:17.732957 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:17.732351    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:17.732957 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:17.732504    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:17.999438 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:17.999242    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" event={"ID":"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1","Type":"ContainerStarted","Data":"063ede657547cc7adf642c82949dc60bc12a58dd00d020ce343603c58da62472"}
Nov 03 19:39:17.999637 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:17.999481    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:39:17.999637 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:17.999514    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:39:18.015394 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:18.015367    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:39:18.077002 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:18.076946    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5" podStartSLOduration=7.022868977 podStartE2EDuration="22.076934411s" totalImagesPullingTime="15.054065434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:18.075567727 +0000 UTC m=+21.825668321" watchObservedRunningTime="2026-11-03 19:39:18.076934411 +0000 UTC m=+21.827035004"
Nov 03 19:39:18.437595 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:18.437500    2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:39:18.438039 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:18.438024    2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:39:18.538290 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:18.538264    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:18.705986 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:18.705903    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:18.732956 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:18.732929    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:18.733487 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:18.733048    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:19.002142 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.002112    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:39:19.019183 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.019152    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:39:19.313502 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.313421    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bxkbj"]
Nov 03 19:39:19.313687 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.313528    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:19.313687 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:19.313638    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:19.317365 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.317334    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2g52t"]
Nov 03 19:39:19.317500 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.317415    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79zbh"]
Nov 03 19:39:19.317544 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.317507    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:19.317665 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:19.317643    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:19.317723 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:19.317710    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:19.317815 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:19.317796    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:20.019258 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:20.019226    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:20.134637 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:20.134603    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:20.678729 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:20.678699    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:20.733133 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:20.733096    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:20.733133 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:20.733117    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:20.733133 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:20.733128    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:20.733383 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:20.733232    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:20.733383 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:20.733350    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:20.733467 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:20.733428    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:20.778965 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:20.778930    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:21.682578 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:21.682547    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:21.792643 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:21.792607    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:22.732211 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:22.732180    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:22.732564 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:22.732227    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:22.732564 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:22.732181    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:22.732564 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:22.732325    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:22.732564 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:22.732372    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:22.732564 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:22.732475    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:23.725397 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:23.725338    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:23.842649 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:23.842612    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:24.173848 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.173815    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:24.316835 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.316800    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:24.436157 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.436047    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:24.732877 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.732849    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:24.732877 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.732861    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:24.733078 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.732869    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:24.733078 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:24.732980    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:24.733157 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:24.733091    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:24.733157 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:24.733147    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:24.910135 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.910107    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:39:24.910608 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.910275    2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:39:24.911011 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:24.910979    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:39:25.074915 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:25.074852    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:25.211073 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:25.211046    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:25.459354 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:25.459271    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:25.620455 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:25.620421    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:25.773775 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:25.773741    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:26.175964 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:26.175892    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:26.734694 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:26.734671    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:26.734845 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:26.734671    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:26.734845 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:26.734765    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:26.734845 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:26.734821    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:26.734845 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:26.734671    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:26.734974 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:26.734913    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:26.766012 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:26.765984    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:27.390081 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:27.390044    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:27.569741 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:27.569707    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:27.973957 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:27.973924    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:28.176867 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.176834    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:28.376092 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.376065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:28.514328 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.514282    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:28.732477 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.732452    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:28.732622 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.732513    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:28.732622 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.732519    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:28.732622 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:28.732588    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:28.732730 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:28.732643    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:28.732730 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:28.732715    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:28.785006 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.784971    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:28.898565 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:28.898529    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:29.288707 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:29.288674    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:29.405721 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:29.405638    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:29.405855 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.405789    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:29.405855 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.405853    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs podName:b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41 nodeName:}" failed. No retries permitted until 2026-11-03 19:40:01.405836464 +0000 UTC m=+65.155937041 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs") pod "network-metrics-daemon-bxkbj" (UID: "b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:39:29.506878 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:29.506842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:29.507057 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:29.506898    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:29.507057 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.506996    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:39:29.507057 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.507017    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:39:29.507057 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.507026    2585 projected.go:196] Error preparing data for projected volume kube-api-access-tzkk5 for pod openshift-network-diagnostics/network-check-target-79zbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:29.507057 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.506998    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:29.507292 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.507071    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5 podName:00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee nodeName:}" failed. No retries permitted until 2026-11-03 19:40:01.507056659 +0000 UTC m=+65.257157245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tzkk5" (UniqueName: "kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5") pod "network-check-target-79zbh" (UID: "00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:39:29.507292 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:29.507115    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret podName:6b3aec1c-dcd1-4b89-86cc-898550befdad nodeName:}" failed. No retries permitted until 2026-11-03 19:40:01.50710178 +0000 UTC m=+65.257202363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret") pod "global-pull-secret-syncer-2g52t" (UID: "6b3aec1c-dcd1-4b89-86cc-898550befdad") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:39:29.798284 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:29.798255    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjqb6_ce3c36fc-764d-4dc1-a515-000a6407542a/dns-node-resolver/0.log"
Nov 03 19:39:30.732549 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:30.732522    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:30.732549 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:30.732539    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:30.732747 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:30.732524    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:30.732747 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:30.732624    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:30.732747 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:30.732684    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:30.732840 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:30.732753    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:30.998028 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:30.997961    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-km679_e5ef855a-3c98-47fd-8559-bb40ad4db198/node-ca/0.log"
Nov 03 19:39:32.732163 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:32.732122    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:32.732649 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:32.732182    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:32.732649 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:32.732217    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:32.732649 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:32.732275    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:32.732649 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:32.732391    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:32.732649 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:32.732464    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:34.400873 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:34.400841    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:34.502775 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:34.502738    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:34.732733 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:34.732704    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:34.732733 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:34.732728    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:34.732971 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:34.732704    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:34.732971 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:34.732829    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:34.732971 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:34.732900    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:34.733122 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:34.732998    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:34.778017 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:34.777981    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:35.379362 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:35.379330    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:35.564656 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:35.564623    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:35.969853 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:35.969818    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:36.170100 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:36.170072    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:36.567535 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:36.567489    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:36.733486 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:36.733452    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:36.733672 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:36.733581    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:36.733672 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:36.733627    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:36.733783 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:36.733692    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:36.733783 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:36.733772    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:36.733889 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:36.733795    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:36.758611 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:36.758566    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:38.732386 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:38.732352    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:38.732869 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:38.732355    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:38.732869 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:38.732461    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:38.732869 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:38.732352    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:38.732869 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:38.732543    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:38.732869 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:38.732608    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:40.732941 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:40.732898    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:40.732941 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:40.732925    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:40.733464 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:40.732940    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:40.733464 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:40.733016    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:40.733464 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:40.733072    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:40.733464 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:40.733171    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:41.127982 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:41.127894    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:41.243924 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:41.243884    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:42.733027 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:42.732986    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:42.733490 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:42.732986    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:42.733490 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:42.733106    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:42.733490 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:42.733222    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:42.733490 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:42.733001    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:42.733490 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:42.733350    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:43.713016 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:43.712983    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:43.825579 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:43.825547    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:44.732626 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:44.732591    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:44.732626 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:44.732613    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:44.732626 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:44.732591    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:44.732936 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:44.732717    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:44.732936 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:44.732792    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:44.732936 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:44.732846    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:46.250650 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.250615    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:46.513364 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.513280    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:46.626851 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.626809    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:46.734867 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.734828    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:46.735053 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.734941    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:46.735053 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:46.734962    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g52t" podUID="6b3aec1c-dcd1-4b89-86cc-898550befdad"
Nov 03 19:39:46.735053 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.735027    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:46.735224 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:46.735023    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxkbj" podUID="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41"
Nov 03 19:39:46.735224 ip-10-0-142-244 kubenswrapper[2585]: E1103 19:39:46.735114    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79zbh" podUID="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee"
Nov 03 19:39:46.744930 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.744894    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:46.953302 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:46.953211    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:47.579745 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.579671    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:39:47.580183 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.579790    2585 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 19:39:47.594916 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.594887    2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:39:47.634659 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.634625    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8fqbg"]
Nov 03 19:39:47.637586 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.637569    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.653455 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.653426    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bv4d2"]
Nov 03 19:39:47.655631 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.655605    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:39:47.656318 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.656293    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:39:47.656430 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.656291    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tlzjx\""
Nov 03 19:39:47.656605 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.656589    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fqbg"]
Nov 03 19:39:47.656696 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.656685    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.663950 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.663927    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-st6kb"]
Nov 03 19:39:47.666716 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.666698    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:47.671254 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.671224    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bv4d2"]
Nov 03 19:39:47.677844 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.677818    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-st6kb"]
Nov 03 19:39:47.737082 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.737048    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43f9158b-9674-477d-a64f-f19fe00bed5b-metrics-tls\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.737082 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.737085    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f9158b-9674-477d-a64f-f19fe00bed5b-tmp-dir\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.737327 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.737111    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r458l\" (UniqueName: \"kubernetes.io/projected/43f9158b-9674-477d-a64f-f19fe00bed5b-kube-api-access-r458l\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.737327 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.737283    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43f9158b-9674-477d-a64f-f19fe00bed5b-config-volume\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.756103 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.756080    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:39:47.756189 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.756172    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:39:47.756253 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.756240    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:39:47.756301 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.756290    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:39:47.765785 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.765763    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:39:47.765868 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.765763    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:39:47.765922 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.765866    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mvxmh\""
Nov 03 19:39:47.765922 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.765885    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hnqg\""
Nov 03 19:39:47.765922 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.765901    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:39:47.838240 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838154    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43f9158b-9674-477d-a64f-f19fe00bed5b-metrics-tls\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.838240 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f9158b-9674-477d-a64f-f19fe00bed5b-tmp-dir\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.838240 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838236    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9af4d64-b784-4e51-aa42-038905055202-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.838414 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838258    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r458l\" (UniqueName: \"kubernetes.io/projected/43f9158b-9674-477d-a64f-f19fe00bed5b-kube-api-access-r458l\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.838414 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838306    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9af4d64-b784-4e51-aa42-038905055202-data-volume\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.838414 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838325    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bb8d09d-f69e-4920-a257-0c957bebd264-cert\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:47.838414 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838340    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9af4d64-b784-4e51-aa42-038905055202-crio-socket\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.838557 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43f9158b-9674-477d-a64f-f19fe00bed5b-config-volume\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.838604 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838584    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjs2\" (UniqueName: \"kubernetes.io/projected/a9af4d64-b784-4e51-aa42-038905055202-kube-api-access-npjs2\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.838683 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838651    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f9158b-9674-477d-a64f-f19fe00bed5b-tmp-dir\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.838740 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838685    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9af4d64-b784-4e51-aa42-038905055202-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.838850 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.838824    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm9f8\" (UniqueName: \"kubernetes.io/projected/6bb8d09d-f69e-4920-a257-0c957bebd264-kube-api-access-qm9f8\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:47.840516 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.840492    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43f9158b-9674-477d-a64f-f19fe00bed5b-config-volume\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.841859 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.841840    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43f9158b-9674-477d-a64f-f19fe00bed5b-metrics-tls\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.847113 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.847085    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r458l\" (UniqueName: \"kubernetes.io/projected/43f9158b-9674-477d-a64f-f19fe00bed5b-kube-api-access-r458l\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.939693 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.939643    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9f8\" (UniqueName: \"kubernetes.io/projected/6bb8d09d-f69e-4920-a257-0c957bebd264-kube-api-access-qm9f8\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:47.939693 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.939686    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9af4d64-b784-4e51-aa42-038905055202-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.939960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.939724    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9af4d64-b784-4e51-aa42-038905055202-data-volume\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.939960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.939742    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bb8d09d-f69e-4920-a257-0c957bebd264-cert\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:47.939960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.939758    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9af4d64-b784-4e51-aa42-038905055202-crio-socket\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.939960 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.939780    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjs2\" (UniqueName: \"kubernetes.io/projected/a9af4d64-b784-4e51-aa42-038905055202-kube-api-access-npjs2\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.940169 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.939968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9af4d64-b784-4e51-aa42-038905055202-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.940169 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.940008    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9af4d64-b784-4e51-aa42-038905055202-crio-socket\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.940169 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.940117    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9af4d64-b784-4e51-aa42-038905055202-data-volume\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.940499 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.940470    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9af4d64-b784-4e51-aa42-038905055202-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.942348 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.942330    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bb8d09d-f69e-4920-a257-0c957bebd264-cert\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:47.942446 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.942382    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9af4d64-b784-4e51-aa42-038905055202-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.948074 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.948044    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9f8\" (UniqueName: \"kubernetes.io/projected/6bb8d09d-f69e-4920-a257-0c957bebd264-kube-api-access-qm9f8\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:47.948179 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.948108    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjs2\" (UniqueName: \"kubernetes.io/projected/a9af4d64-b784-4e51-aa42-038905055202-kube-api-access-npjs2\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.950412 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.950392    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:47.973148 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.973118    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:39:47.979186 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:47.979161    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:39:48.248114 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.247925    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fqbg"]
Nov 03 19:39:48.251026 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:39:48.251003    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f9158b_9674_477d_a64f_f19fe00bed5b.slice/crio-0135e72e24b7928925edb15d299588c77ffe5456071c1ecdcfeb8ffaf69da267 WatchSource:0}: Error finding container 0135e72e24b7928925edb15d299588c77ffe5456071c1ecdcfeb8ffaf69da267: Status 404 returned error can't find the container with id 0135e72e24b7928925edb15d299588c77ffe5456071c1ecdcfeb8ffaf69da267
Nov 03 19:39:48.296595 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.296561    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-st6kb"]
Nov 03 19:39:48.300501 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:39:48.300391    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb8d09d_f69e_4920_a257_0c957bebd264.slice/crio-d43aa2153a3723f97993b80cee8c6b450a7a944e2c290c33ce32f9ff782cb146 WatchSource:0}: Error finding container d43aa2153a3723f97993b80cee8c6b450a7a944e2c290c33ce32f9ff782cb146: Status 404 returned error can't find the container with id d43aa2153a3723f97993b80cee8c6b450a7a944e2c290c33ce32f9ff782cb146
Nov 03 19:39:48.300577 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.300554    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bv4d2"]
Nov 03 19:39:48.302851 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:39:48.302828    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9af4d64_b784_4e51_aa42_038905055202.slice/crio-4fc07e2911af26dec136f6d483ac02be64dbc278231996d1046e713a2cc53712 WatchSource:0}: Error finding container 4fc07e2911af26dec136f6d483ac02be64dbc278231996d1046e713a2cc53712: Status 404 returned error can't find the container with id 4fc07e2911af26dec136f6d483ac02be64dbc278231996d1046e713a2cc53712
Nov 03 19:39:48.520305 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.520226    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"]
Nov 03 19:39:48.523233 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.523217    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.527934 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.527903    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 19:39:48.531464 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.531445    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:39:48.534261 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.534244    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:39:48.534381 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.534349    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:39:48.535639 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.535617    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"]
Nov 03 19:39:48.544742 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.544692    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:48.558297 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.558270    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 19:39:48.558425 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.558323    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9dv56\""
Nov 03 19:39:48.645026 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.644987    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.645419 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.645056    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/076c5bac-22d5-4145-9dd6-bfc03e1b858e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.645419 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.645167    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.645419 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.645193    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxrd\" (UniqueName: \"kubernetes.io/projected/076c5bac-22d5-4145-9dd6-bfc03e1b858e-kube-api-access-hwxrd\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.654110 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.654070    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:48.732288 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.732248    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:39:48.732457 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.732343    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:39:48.732522 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.732461    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:39:48.736732 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.736656    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:39:48.736878 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.736655    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:39:48.736878 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.736684    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\""
Nov 03 19:39:48.736878 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.736748    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\""
Nov 03 19:39:48.736878 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.736788    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:39:48.736878 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.736813    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\""
Nov 03 19:39:48.737519 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.737504    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:39:48.746330 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.746304    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/076c5bac-22d5-4145-9dd6-bfc03e1b858e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.746461 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.746375    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.746461 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.746401    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxrd\" (UniqueName: \"kubernetes.io/projected/076c5bac-22d5-4145-9dd6-bfc03e1b858e-kube-api-access-hwxrd\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.746461 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.746438    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.748213 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.747337    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/076c5bac-22d5-4145-9dd6-bfc03e1b858e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.749685 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.749499    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.750896 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.750870    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.754385 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.754363    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxrd\" (UniqueName: \"kubernetes.io/projected/076c5bac-22d5-4145-9dd6-bfc03e1b858e-kube-api-access-hwxrd\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:48.836884 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:48.836796    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:39:49.064679 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:49.064644    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-st6kb" event={"ID":"6bb8d09d-f69e-4920-a257-0c957bebd264","Type":"ContainerStarted","Data":"d43aa2153a3723f97993b80cee8c6b450a7a944e2c290c33ce32f9ff782cb146"}
Nov 03 19:39:49.066707 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:49.066676    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv4d2" event={"ID":"a9af4d64-b784-4e51-aa42-038905055202","Type":"ContainerStarted","Data":"a31791a782eed2f1a63c3f82f59a031123a8329f7047ae7b0cdbd3fd0f817a99"}
Nov 03 19:39:49.066802 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:49.066729    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv4d2" event={"ID":"a9af4d64-b784-4e51-aa42-038905055202","Type":"ContainerStarted","Data":"4fc07e2911af26dec136f6d483ac02be64dbc278231996d1046e713a2cc53712"}
Nov 03 19:39:49.069425 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:49.069395    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fqbg" event={"ID":"43f9158b-9674-477d-a64f-f19fe00bed5b","Type":"ContainerStarted","Data":"0135e72e24b7928925edb15d299588c77ffe5456071c1ecdcfeb8ffaf69da267"}
Nov 03 19:39:49.118875 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:49.118837    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"]
Nov 03 19:39:49.123897 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:39:49.123860    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076c5bac_22d5_4145_9dd6_bfc03e1b858e.slice/crio-19af5aa458d02ed4a0ee30f80b45f9a57688cd33fb59feb49e740adcba0f6109 WatchSource:0}: Error finding container 19af5aa458d02ed4a0ee30f80b45f9a57688cd33fb59feb49e740adcba0f6109: Status 404 returned error can't find the container with id 19af5aa458d02ed4a0ee30f80b45f9a57688cd33fb59feb49e740adcba0f6109
Nov 03 19:39:50.076383 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:50.076086    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv4d2" event={"ID":"a9af4d64-b784-4e51-aa42-038905055202","Type":"ContainerStarted","Data":"413614ecf37ace3eabed4b5f2e4205e435df8a129104bed4082cf73ba23e5702"}
Nov 03 19:39:50.078180 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:50.078149    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h" event={"ID":"076c5bac-22d5-4145-9dd6-bfc03e1b858e","Type":"ContainerStarted","Data":"19af5aa458d02ed4a0ee30f80b45f9a57688cd33fb59feb49e740adcba0f6109"}
Nov 03 19:39:51.018886 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.018856    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:39:51.083419 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.083369    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h" event={"ID":"076c5bac-22d5-4145-9dd6-bfc03e1b858e","Type":"ContainerStarted","Data":"13025ad22395d42f995e6fe37f77eb25464d9dda225241a78a162801f8319b91"}
Nov 03 19:39:51.083864 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.083439    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h" event={"ID":"076c5bac-22d5-4145-9dd6-bfc03e1b858e","Type":"ContainerStarted","Data":"26acbd93582ba760fd2382327eae332d26a121756d37173b38f77bcb3ca6593d"}
Nov 03 19:39:51.087120 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.086760    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fqbg" event={"ID":"43f9158b-9674-477d-a64f-f19fe00bed5b","Type":"ContainerStarted","Data":"e588e273d37ce40adf76e42d7361f5e31a172bdbf823f83f9be30a1f27f15cda"}
Nov 03 19:39:51.087120 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.086823    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fqbg" event={"ID":"43f9158b-9674-477d-a64f-f19fe00bed5b","Type":"ContainerStarted","Data":"5a096d5ae83307b436cd5d3ea884305b381db5bb2fa531b795bf160093e19d83"}
Nov 03 19:39:51.087120 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.086856    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:39:51.089373 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.089309    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv4d2" event={"ID":"a9af4d64-b784-4e51-aa42-038905055202","Type":"ContainerStarted","Data":"e0be6bf14f29b7bafd427f91c1350431fa2be323230ed7ed8b253515c7601d2d"}
Nov 03 19:39:51.103597 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.103547    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h" podStartSLOduration=1.37615048 podStartE2EDuration="3.103532521s" totalImagesPullingTime="1.727382041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:51.102577681 +0000 UTC m=+54.852678275" watchObservedRunningTime="2026-11-03 19:39:51.103532521 +0000 UTC m=+54.853633108"
Nov 03 19:39:51.147045 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.146897    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8fqbg" podStartSLOduration=2.15382537 podStartE2EDuration="4.146881845s" totalImagesPullingTime="1.993056475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:51.125246212 +0000 UTC m=+54.875346809" watchObservedRunningTime="2026-11-03 19:39:51.146881845 +0000 UTC m=+54.896982441"
Nov 03 19:39:51.147045 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.147029    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bv4d2" podStartSLOduration=1.6954810679999999 podStartE2EDuration="4.147020708s" totalImagesPullingTime="2.45153964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:51.146277823 +0000 UTC m=+54.896378418" watchObservedRunningTime="2026-11-03 19:39:51.147020708 +0000 UTC m=+54.897121305"
Nov 03 19:39:51.218567 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:51.218526    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:52.094386 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.094348    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-st6kb" event={"ID":"6bb8d09d-f69e-4920-a257-0c957bebd264","Type":"ContainerStarted","Data":"5873f686988919daf342cac5e9e84856418a6aa16d25d3a42204fc1144099e61"}
Nov 03 19:39:52.110972 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.110922    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-st6kb" podStartSLOduration=2.036622772 podStartE2EDuration="5.110908942s" totalImagesPullingTime="3.07428617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:52.110288993 +0000 UTC m=+55.860389585" watchObservedRunningTime="2026-11-03 19:39:52.110908942 +0000 UTC m=+55.861009536"
Nov 03 19:39:52.683473 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.683431    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:52.854044 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.854005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:52.926102 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.926070    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b2qlc"]
Nov 03 19:39:52.929397 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.929379    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981269 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981438 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981274    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-textfile\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981438 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981438 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981425    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-sys\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981555 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-tls\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981555 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981494    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-wtmp\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981555 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-root\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981692 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981653    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-metrics-client-ca\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.981747 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.981696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757sg\" (UniqueName: \"kubernetes.io/projected/48f77550-ff58-46e6-b7d8-6840e4afa52c-kube-api-access-757sg\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:52.984061 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:52.984038    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:53.013969 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.013936    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:39:53.013969 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.013962    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g7dvr\""
Nov 03 19:39:53.014182 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.014165    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:39:53.016866 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.016848    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:39:53.082445 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082404    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082588 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082451    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-sys\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082588 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082479    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-tls\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082588 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082500    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-wtmp\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082588 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082536    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-sys\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082588 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-root\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082588 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082584    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-root\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082957 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082643    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-metrics-client-ca\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082957 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082669    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757sg\" (UniqueName: \"kubernetes.io/projected/48f77550-ff58-46e6-b7d8-6840e4afa52c-kube-api-access-757sg\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082957 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082696    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-wtmp\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082957 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082703    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.082957 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.082867    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-textfile\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.083260 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.083233    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-textfile\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.083329 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.083290    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-metrics-client-ca\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.083383 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.083354    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.084907 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.084888    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.085113 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.085097    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-tls\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.091598 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.091573    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-757sg\" (UniqueName: \"kubernetes.io/projected/48f77550-ff58-46e6-b7d8-6840e4afa52c-kube-api-access-757sg\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.128652 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.128617    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:53.244688 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.244657    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:39:53.257712 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:39:53.257680    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f77550_ff58_46e6_b7d8_6840e4afa52c.slice/crio-4b0306d8e3bbdeb835e588ceff000b76040076c0b1fbf4d7169182f51f39b3e2 WatchSource:0}: Error finding container 4b0306d8e3bbdeb835e588ceff000b76040076c0b1fbf4d7169182f51f39b3e2: Status 404 returned error can't find the container with id 4b0306d8e3bbdeb835e588ceff000b76040076c0b1fbf4d7169182f51f39b3e2
Nov 03 19:39:53.342097 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.342051    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:53.494159 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.494120    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:53.924956 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:53.924913    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:54.102414 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.102382    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2qlc" event={"ID":"48f77550-ff58-46e6-b7d8-6840e4afa52c","Type":"ContainerStarted","Data":"4b0306d8e3bbdeb835e588ceff000b76040076c0b1fbf4d7169182f51f39b3e2"}
Nov 03 19:39:54.131883 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.131849    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:54.979863 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.979827    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"]
Nov 03 19:39:54.983100 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.983082    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:54.986849 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.986825    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 19:39:54.988149 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.988100    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 19:39:54.988149 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.988119    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 19:39:54.988343 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.988158    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 19:39:54.988343 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.988170    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d2i8be8p11s16\""
Nov 03 19:39:54.988343 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.988213    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 19:39:54.988343 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:54.988278    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-67k2x\""
Nov 03 19:39:55.001794 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.001760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002001 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.001828    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a113235-00b3-42db-ac16-0fc01da5a788-metrics-client-ca\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002001 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.001894    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"]
Nov 03 19:39:55.002001 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.001934    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002001 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.001995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a113235-00b3-42db-ac16-0fc01da5a788-tmp\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.002151    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8gj\" (UniqueName: \"kubernetes.io/projected/9a113235-00b3-42db-ac16-0fc01da5a788-kube-api-access-nq8gj\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.002219    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002287 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.002269    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002447 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.002302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-grpc-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.002447 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.002338    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102737 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102703    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8gj\" (UniqueName: \"kubernetes.io/projected/9a113235-00b3-42db-ac16-0fc01da5a788-kube-api-access-nq8gj\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102943 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102745    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102943 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102775    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102943 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102791    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-grpc-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102943 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102943 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102943 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102893    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a113235-00b3-42db-ac16-0fc01da5a788-metrics-client-ca\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.102943 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102925    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.103280 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.102967    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a113235-00b3-42db-ac16-0fc01da5a788-tmp\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.103607 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.103579    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a113235-00b3-42db-ac16-0fc01da5a788-tmp\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.103824 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.103799    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a113235-00b3-42db-ac16-0fc01da5a788-metrics-client-ca\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.106253 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.106226    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.106382 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.106230    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.106382 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.106284    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.106382 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.106346    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.106382 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.106349    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.106561 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.106288    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-grpc-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.108103 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.108078    2585 generic.go:356] "Generic (PLEG): container finished" podID="48f77550-ff58-46e6-b7d8-6840e4afa52c" containerID="59ffad534822c625910e487cfac976badf8497e1bf87423e17deb7a826d05d81" exitCode=0
Nov 03 19:39:55.108226 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.108115    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2qlc" event={"ID":"48f77550-ff58-46e6-b7d8-6840e4afa52c","Type":"ContainerDied","Data":"59ffad534822c625910e487cfac976badf8497e1bf87423e17deb7a826d05d81"}
Nov 03 19:39:55.111062 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.111046    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8gj\" (UniqueName: \"kubernetes.io/projected/9a113235-00b3-42db-ac16-0fc01da5a788-kube-api-access-nq8gj\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.307246 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.307120    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:55.460640 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.460603    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:55.583274 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.583182    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"]
Nov 03 19:39:55.586110 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:39:55.586076    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a113235_00b3_42db_ac16_0fc01da5a788.slice/crio-094ea790cbff5d2352269947c0c100c2ba7258c7cbf5150a7bef9af999e30895 WatchSource:0}: Error finding container 094ea790cbff5d2352269947c0c100c2ba7258c7cbf5150a7bef9af999e30895: Status 404 returned error can't find the container with id 094ea790cbff5d2352269947c0c100c2ba7258c7cbf5150a7bef9af999e30895
Nov 03 19:39:55.885975 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:55.885850    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:56.015221 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:56.015143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:56.114129 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:56.114091    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2qlc" event={"ID":"48f77550-ff58-46e6-b7d8-6840e4afa52c","Type":"ContainerStarted","Data":"6a48f3a056735c3fa7b6a9f422d5a577473227acfaaa48ef5701a7b264fb2bfd"}
Nov 03 19:39:56.114319 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:56.114156    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b2qlc" event={"ID":"48f77550-ff58-46e6-b7d8-6840e4afa52c","Type":"ContainerStarted","Data":"eccb3154157ce1c09c5b46bd33b39c151d7b2bfc5bbe4adead890b69464d7861"}
Nov 03 19:39:56.115250 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:56.115220    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" event={"ID":"9a113235-00b3-42db-ac16-0fc01da5a788","Type":"ContainerStarted","Data":"094ea790cbff5d2352269947c0c100c2ba7258c7cbf5150a7bef9af999e30895"}
Nov 03 19:39:56.134545 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:56.134494    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b2qlc" podStartSLOduration=2.982359337 podStartE2EDuration="4.134480254s" totalImagesPullingTime="1.084040755s" totalInitContainerRuntime="68.080162ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:56.133881798 +0000 UTC m=+59.883982431" watchObservedRunningTime="2026-11-03 19:39:56.134480254 +0000 UTC m=+59.884580847"
Nov 03 19:39:56.150186 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:56.150102    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:57.301803 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.301756    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:57.435756 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.435721    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:57.697241 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.696264    2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"]
Nov 03 19:39:57.699334 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.699309    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:57.703038 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.703015    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4j9gs\""
Nov 03 19:39:57.703038 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.703039    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 19:39:57.710986 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.710951    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"]
Nov 03 19:39:57.724375 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.724246    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp96d\" (UniqueName: \"kubernetes.io/projected/58e236c2-f2bc-4adc-8daf-f160c4c882f1-kube-api-access-fp96d\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:57.724375 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.724319    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58e236c2-f2bc-4adc-8daf-f160c4c882f1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:57.733303 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.733258    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:57.825807 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.825715    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp96d\" (UniqueName: \"kubernetes.io/projected/58e236c2-f2bc-4adc-8daf-f160c4c882f1-kube-api-access-fp96d\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:57.825807 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.825775    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58e236c2-f2bc-4adc-8daf-f160c4c882f1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:57.828284 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.828256    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58e236c2-f2bc-4adc-8daf-f160c4c882f1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:57.834021 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.833997    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp96d\" (UniqueName: \"kubernetes.io/projected/58e236c2-f2bc-4adc-8daf-f160c4c882f1-kube-api-access-fp96d\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:57.875890 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:57.875855    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:58.013833 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:58.013799    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:39:58.124072 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:58.123981    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" event={"ID":"9a113235-00b3-42db-ac16-0fc01da5a788","Type":"ContainerStarted","Data":"44145a4f679e7594e3f00224699ebc811c285bdcae89ab1d62142355a67e6e66"}
Nov 03 19:39:58.124072 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:58.124058    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" event={"ID":"9a113235-00b3-42db-ac16-0fc01da5a788","Type":"ContainerStarted","Data":"5ff8dc6ae9bcfc15c4679ec4985b5aa871bb82359cef3312b94624be3af59ba3"}
Nov 03 19:39:58.124308 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:58.124137    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" event={"ID":"9a113235-00b3-42db-ac16-0fc01da5a788","Type":"ContainerStarted","Data":"0bce380472b402bef2ace0afe66e53eafe674b51fdf3c24e8bf82597cbcca14f"}
Nov 03 19:39:58.285575 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:58.285543    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"]
Nov 03 19:39:58.288781 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:39:58.288746    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e236c2_f2bc_4adc_8daf_f160c4c882f1.slice/crio-401f75d98667e3ca023238787e853a4d3b72e59ccb780851c0a38866535bfa49 WatchSource:0}: Error finding container 401f75d98667e3ca023238787e853a4d3b72e59ccb780851c0a38866535bfa49: Status 404 returned error can't find the container with id 401f75d98667e3ca023238787e853a4d3b72e59ccb780851c0a38866535bfa49
Nov 03 19:39:59.136454 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.136411    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" event={"ID":"9a113235-00b3-42db-ac16-0fc01da5a788","Type":"ContainerStarted","Data":"cc830530bb6b052954835fb2d0ee63d4e268de4431f54acc700f556358ba42ce"}
Nov 03 19:39:59.136945 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.136503    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" event={"ID":"9a113235-00b3-42db-ac16-0fc01da5a788","Type":"ContainerStarted","Data":"1b8ff8c7d27e2885e569ece0c3b7f037542148ab66b292dd3b4e47f43687fc5a"}
Nov 03 19:39:59.136945 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.136549    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" event={"ID":"9a113235-00b3-42db-ac16-0fc01da5a788","Type":"ContainerStarted","Data":"e542bafe6257cd0dac94862c793e5588b0b810d7324e7f66e98a225caa9cb686"}
Nov 03 19:39:59.136945 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.136686    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:39:59.138171 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.138142    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5" event={"ID":"58e236c2-f2bc-4adc-8daf-f160c4c882f1","Type":"ContainerStarted","Data":"401f75d98667e3ca023238787e853a4d3b72e59ccb780851c0a38866535bfa49"}
Nov 03 19:39:59.168934 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.167265    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" podStartSLOduration=2.437251454 podStartE2EDuration="5.167250001s" totalImagesPullingTime="2.729998547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:59.164392196 +0000 UTC m=+62.914492787" watchObservedRunningTime="2026-11-03 19:39:59.167250001 +0000 UTC m=+62.917350599"
Nov 03 19:39:59.192868 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.192834    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:59.659800 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.659770    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:59.773152 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:39:59.773123    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:00.143654 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:00.143608    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5" event={"ID":"58e236c2-f2bc-4adc-8daf-f160c4c882f1","Type":"ContainerStarted","Data":"f606d139771756156fae8eaa103ddc33c2fcb89c7cf060efde3ffbf96bb56cb8"}
Nov 03 19:40:00.144123 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:00.143825    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:00.148662 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:00.148634    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:00.159780 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:00.159720    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5" podStartSLOduration=1.724166037 podStartE2EDuration="3.159704323s" totalImagesPullingTime="1.435538286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:00.158993697 +0000 UTC m=+63.909094294" watchObservedRunningTime="2026-11-03 19:40:00.159704323 +0000 UTC m=+63.909804917"
Nov 03 19:40:01.096839 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.096806    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:01.457644 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.457562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:40:01.460525 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.460503    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:40:01.470926 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.470899    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:40:01.558402 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.558363    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:01.558606 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.558429    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:01.561183 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.561159    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:40:01.561249 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.561167    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:40:01.571606 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.571580    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:01.572043 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.572029    2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:40:01.582527 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.582494    2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:01.651334 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.651303    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\""
Nov 03 19:40:01.656817 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.656795    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\""
Nov 03 19:40:01.658880 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.658857    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:40:01.662571 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.662542    2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\""
Nov 03 19:40:01.664930 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.664914    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:01.671057 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.671024    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:01.936874 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.936828    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bxkbj"]
Nov 03 19:40:01.939709 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:40:01.939677    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ca8e71_a6ff_48a6_b58c_fdd2bbbc3c41.slice/crio-6a4fe582af93d4a63d2d2e433745d7b3b1d58d0b5e916e4d8e4549166e33e735 WatchSource:0}: Error finding container 6a4fe582af93d4a63d2d2e433745d7b3b1d58d0b5e916e4d8e4549166e33e735: Status 404 returned error can't find the container with id 6a4fe582af93d4a63d2d2e433745d7b3b1d58d0b5e916e4d8e4549166e33e735
Nov 03 19:40:01.939828 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.939757    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2g52t"]
Nov 03 19:40:01.944351 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:40:01.944329    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3aec1c_dcd1_4b89_86cc_898550befdad.slice/crio-9b8cb613554626339b48afa2136aff35fa3fc728e9ef6b1ec608e0f9f4e4c2c1 WatchSource:0}: Error finding container 9b8cb613554626339b48afa2136aff35fa3fc728e9ef6b1ec608e0f9f4e4c2c1: Status 404 returned error can't find the container with id 9b8cb613554626339b48afa2136aff35fa3fc728e9ef6b1ec608e0f9f4e4c2c1
Nov 03 19:40:01.958336 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:01.958271    2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79zbh"]
Nov 03 19:40:01.960772 ip-10-0-142-244 kubenswrapper[2585]: W1103 19:40:01.960749    2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a10aa1_fa7a_4b73_9ffd_ca212b2b20ee.slice/crio-8d69387f8beb09e445d317a17b6a9059df08bfa4a4f1180d1318029e8ffd0850 WatchSource:0}: Error finding container 8d69387f8beb09e445d317a17b6a9059df08bfa4a4f1180d1318029e8ffd0850: Status 404 returned error can't find the container with id 8d69387f8beb09e445d317a17b6a9059df08bfa4a4f1180d1318029e8ffd0850
Nov 03 19:40:02.151052 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:02.151016    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79zbh" event={"ID":"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee","Type":"ContainerStarted","Data":"8d69387f8beb09e445d317a17b6a9059df08bfa4a4f1180d1318029e8ffd0850"}
Nov 03 19:40:02.152118 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:02.152094    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2g52t" event={"ID":"6b3aec1c-dcd1-4b89-86cc-898550befdad","Type":"ContainerStarted","Data":"9b8cb613554626339b48afa2136aff35fa3fc728e9ef6b1ec608e0f9f4e4c2c1"}
Nov 03 19:40:02.153225 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:02.153190    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxkbj" event={"ID":"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41","Type":"ContainerStarted","Data":"6a4fe582af93d4a63d2d2e433745d7b3b1d58d0b5e916e4d8e4549166e33e735"}
Nov 03 19:40:04.165326 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:04.165273    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxkbj" event={"ID":"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41","Type":"ContainerStarted","Data":"450118347e9420d83cb72bf44a0df674bc1773260fccaf1b98972228d9da4ffd"}
Nov 03 19:40:04.165998 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:04.165347    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxkbj" event={"ID":"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41","Type":"ContainerStarted","Data":"152a15986b0400e7ec9091d3a436fcf31da00b041873fec8948347662d904ea1"}
Nov 03 19:40:04.193260 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:04.193153    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bxkbj" podStartSLOduration=66.608901002 podStartE2EDuration="1m8.193130004s" totalImagesPullingTime="1.584229002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:04.191124582 +0000 UTC m=+67.941225177" watchObservedRunningTime="2026-11-03 19:40:04.193130004 +0000 UTC m=+67.943230599"
Nov 03 19:40:05.151908 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:05.151876    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:06.175002 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:06.174968    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79zbh" event={"ID":"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee","Type":"ContainerStarted","Data":"e65ecb5d8ea7ac496be9f37508a5907b9ddb7112c9c9ad386130321716612978"}
Nov 03 19:40:06.175504 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:06.175248    2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:06.197414 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:06.197367    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-79zbh" podStartSLOduration=66.683312329 podStartE2EDuration="1m10.197351778s" totalImagesPullingTime="3.514039449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:06.195458072 +0000 UTC m=+69.945558663" watchObservedRunningTime="2026-11-03 19:40:06.197351778 +0000 UTC m=+69.947452374"
Nov 03 19:40:06.449938 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:06.449825    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:06.566083 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:06.566052    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:07.180803 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:07.180760    2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2g52t" event={"ID":"6b3aec1c-dcd1-4b89-86cc-898550befdad","Type":"ContainerStarted","Data":"974166a1c302fadb8930910db9e0aafb89fac458406e6c3ece641f11fdca47a3"}
Nov 03 19:40:07.197017 ip-10-0-142-244 kubenswrapper[2585]: I1103 19:40:07.196963    2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2g52t" podStartSLOduration=66.129632541 podStartE2EDuration="1m11.19694865s" totalImagesPullingTime="5.067316109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:07.196913929 +0000 UTC m=+70.947014524" watchObservedRunningTime="2026-11-03 19:40:07.19694865 +0000 UTC m=+70.947049241"
Nov 03 19:40:07.273939 ip-10-0-142-244 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 19:40:07.282600 ip-10-0-142-244 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 19:40:07.282815 ip-10-0-142-244 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 19:40:07.282880 ip-10-0-142-244 systemd[1]: kubelet.service: Consumed 3.896s CPU time.
Nov 03 19:40:07.299086 ip-10-0-142-244 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:40:07.363856 ip-10-0-142-244 kubenswrapper[5282]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:40:07.363856 ip-10-0-142-244 kubenswrapper[5282]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:40:07.363856 ip-10-0-142-244 kubenswrapper[5282]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:40:07.363856 ip-10-0-142-244 kubenswrapper[5282]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:40:07.366006 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.365915    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.366006 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366007    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366012    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366015    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366018    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366021    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366024    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366027    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366030    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366033    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366036    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366038    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366041    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366044    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366047    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366049    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366052    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366055    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366057    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366060    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.366079 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366063    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366065    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366068    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366072    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366075    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366077    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366080    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366083    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366086    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366089    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366092    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366094    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366097    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366101    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366103    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366106    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366108    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366111    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366113    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366116    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.366563 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366118    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366121    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366123    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366126    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366129    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366132    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366134    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366137    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366140    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366142    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366145    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366147    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366151    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366154    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366159    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366162    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366164    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366167    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366169    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.367053 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366172    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366175    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366178    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366181    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366183    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366186    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366190    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366207    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366212    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366216    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366220    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366223    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366225    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366228    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366231    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366234    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366237    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366239    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366242    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366247    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.367532 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366252    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366255    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366259    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366262    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366265    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366268    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366270    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366273    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366276    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366279    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366282    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366284    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366287    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366290    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366292    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366295    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366297    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366300    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366303    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.368025 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366306    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366311    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366315    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366318    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366321    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366323    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366326    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.366328    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367152    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367160    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367163    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367166    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367169    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367172    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367175    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367178    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367180    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367183    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367186    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.368576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367189    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367192    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367211    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367214    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367217    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367220    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367223    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367226    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367228    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367231    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367234    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367236    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367239    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367242    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367245    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367248    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367251    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367253    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367256    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367259    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.369052 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367261    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367265    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367268    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367270    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367273    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367276    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367279    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367282    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367284    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367287    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367290    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367293    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367295    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367298    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367301    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367303    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367306    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367308    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367311    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367313    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.369557 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367316    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367318    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367321    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367324    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367328    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367333    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367336    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367339    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367343    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367345    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367348    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367355    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367359    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367362    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367365    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367368    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367370    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367373    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367376    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367379    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.370042 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367382    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367384    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367387    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367390    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367392    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367395    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367398    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367401    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367404    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367406    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367409    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367412    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367414    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367417    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367420    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367422    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367425    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367428    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367430    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.370566 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367433    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367442    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367445    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367448    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367451    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367454    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367457    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367459    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367466    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367469    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367472    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367474    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367477    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367481    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367484    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.367486    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368005    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368012    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368016    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.371026 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368019    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368022    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368025    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368027    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368030    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368033    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368036    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368039    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368042    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368045    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368047    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368050    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368052    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368055    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368058    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368061    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368064    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368068    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368070    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368073    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.371515 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368075    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368078    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368080    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368083    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368085    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368088    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368092    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368095    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368097    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368100    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368104    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368107    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368111    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368113    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368117    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368119    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368122    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368125    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368127    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.372015 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368130    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368132    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368135    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368137    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368140    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368142    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368145    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368148    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368151    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368154    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368157    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368159    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368162    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368164    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368167    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368169    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368172    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368183    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368187    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368190    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.372576 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368210    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368214    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368217    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368220    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368222    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368225    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368228    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368231    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368234    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368236    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368239    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368241    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368245    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368248    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368251    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368254    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368256    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368259    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368261    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368264    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.373395 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368266    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368269    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368272    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368275    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368278    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368280    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368283    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368286    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368288    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368291    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368293    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368297    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368300    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368303    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368305    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368308    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368311    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368313    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368316    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.374208 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368319    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368321    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368324    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368327    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.368329    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368432    5282 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368446    5282 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368453    5282 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368458    5282 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368462    5282 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368466    5282 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368470    5282 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368479    5282 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368483    5282 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368486    5282 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368491    5282 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368494    5282 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368498    5282 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368501    5282 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368504    5282 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368507    5282 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368510    5282 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368513    5282 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368517    5282 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:40:07.375018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368522    5282 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368525    5282 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368528    5282 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368531    5282 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368536    5282 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368540    5282 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368545    5282 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368548    5282 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368552    5282 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368555    5282 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368558    5282 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368562    5282 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368565    5282 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368569    5282 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368572    5282 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368576    5282 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368579    5282 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368582    5282 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368586    5282 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368591    5282 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368594    5282 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368597    5282 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368601    5282 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368604    5282 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:40:07.376118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368609    5282 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368612    5282 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368615    5282 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368618    5282 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368621    5282 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368624    5282 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368627    5282 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368630    5282 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368634    5282 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368637    5282 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368640    5282 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368644    5282 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368647    5282 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368651    5282 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368655    5282 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368659    5282 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368662    5282 flags.go:64] FLAG: --help="false"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368665    5282 flags.go:64] FLAG: --hostname-override="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368668    5282 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368672    5282 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368674    5282 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368678    5282 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368681    5282 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:40:07.377285 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368684    5282 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368687    5282 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368690    5282 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368693    5282 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368696    5282 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368699    5282 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368703    5282 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368706    5282 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368709    5282 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368712    5282 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368715    5282 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368719    5282 flags.go:64] FLAG: --lock-file=""
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368722    5282 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368725    5282 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368728    5282 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368734    5282 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368738    5282 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368741    5282 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368744    5282 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368747    5282 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368750    5282 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368753    5282 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368756    5282 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368765    5282 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368769    5282 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:40:07.378281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368773    5282 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368776    5282 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368779    5282 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368782    5282 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368785    5282 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368788    5282 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368791    5282 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368794    5282 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368801    5282 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368805    5282 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368808    5282 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368811    5282 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368814    5282 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368817    5282 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368820    5282 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368823    5282 flags.go:64] FLAG: --port="10250"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368827    5282 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368830    5282 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03af00eb5275d65bf"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368834    5282 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368837    5282 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368841    5282 flags.go:64] FLAG: --register-node="true"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368844    5282 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368848    5282 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368851    5282 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368854    5282 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368857    5282 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:40:07.379432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368861    5282 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368864    5282 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368867    5282 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368870    5282 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368874    5282 flags.go:64] FLAG: --runonce="false"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368877    5282 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368881    5282 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368884    5282 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368887    5282 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368890    5282 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368894    5282 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368897    5282 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368901    5282 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368904    5282 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368907    5282 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368910    5282 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368913    5282 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368916    5282 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368919    5282 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368923    5282 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368928    5282 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368931    5282 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368934    5282 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368938    5282 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368941    5282 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:40:07.380536 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368945    5282 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368948    5282 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368951    5282 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368954    5282 flags.go:64] FLAG: --v="2"
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368959    5282 flags.go:64] FLAG: --version="false"
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368963    5282 flags.go:64] FLAG: --vmodule=""
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368967    5282 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.368970    5282 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369107    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369111    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369114    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369118    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369121    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369125    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369129    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369132    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369135    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369139    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369142    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369145    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369147    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.381578 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369150    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369153    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369158    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369161    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369164    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369166    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369169    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369172    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369174    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369177    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369179    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369183    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369188    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369191    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369208    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369212    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369214    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369217    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369220    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.382439 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369223    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369226    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369228    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369232    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369235    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369237    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369240    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369243    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369245    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369248    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369251    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369253    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369256    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369259    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369261    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369265    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369268    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369271    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369274    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369276    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.382983 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369279    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369281    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369284    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369287    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369289    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369292    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369295    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369298    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369301    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369304    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369306    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369309    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369312    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369314    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369317    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369321    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369324    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369327    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369329    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.383684 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369332    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369335    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369337    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369340    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369342    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369345    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369348    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369350    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369354    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369357    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369360    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369362    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369365    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369367    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369370    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369372    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369375    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369378    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369380    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369383    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.384249 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369386    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369389    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369392    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369394    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369397    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369399    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369402    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369404    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369409    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369411    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369414    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369417    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369420    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369422    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369425    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.369451    5282 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:40:07.384736 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369489    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369494    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369498    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369501    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369505    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369508    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369511    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369514    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369517    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369519    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369522    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369525    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369529    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369532    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369535    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369538    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369541    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369544    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369547    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369550    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.385173 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369552    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369555    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369558    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369561    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369563    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369568    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369570    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369573    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369576    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369578    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369581    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369585    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369589    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369592    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369594    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369597    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369601    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369603    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369606    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.385805 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369609    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369611    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369614    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369616    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369619    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369621    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369624    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369626    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369629    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369632    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369634    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369637    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369640    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369643    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369647    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369649    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369652    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369654    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369658    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369661    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.386354 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369664    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369666    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369669    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369671    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369674    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369676    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369679    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369682    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369684    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369688    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369691    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369694    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369697    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369699    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369702    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369705    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369707    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369710    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369712    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.386888 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369715    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369717    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369720    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369722    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369725    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369728    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369731    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369733    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369736    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369739    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369741    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369745    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369748    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369750    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369753    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369755    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369758    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369761    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369764    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369766    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.387369 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369769    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369771    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369776    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369778    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369781    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369784    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369786    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.369789    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.375221    5282 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.375247    5282 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.375273    5282 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.375281    5282 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375375    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375385    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375390    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375396    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375401    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375407    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375412    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375418    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.387849 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375423    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375428    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375432    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375437    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375441    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375446    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375450    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375462    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375467    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375472    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375476    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375481    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375485    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375490    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375494    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375501    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375507    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375512    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375516    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375520    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.388404 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375525    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375530    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375534    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375539    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375543    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375548    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375552    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375557    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375561    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375565    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375570    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375574    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375578    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375583    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375588    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375592    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375597    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375602    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375606    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.388911 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375611    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375615    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375619    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375623    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375627    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375632    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375636    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375641    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375645    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375650    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375655    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375660    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375664    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375668    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375673    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375677    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375682    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375686    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375691    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375696    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.389394 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375700    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375704    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375709    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375713    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375719    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375724    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375728    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375733    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375737    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375742    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375746    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375750    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375763    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375768    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375772    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375776    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375781    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375785    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375791    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.389864 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375798    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375803    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375808    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375813    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375817    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375822    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375826    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375831    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375836    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375841    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375845    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375849    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375854    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375858    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375863    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375867    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375871    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375876    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375880    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.390511 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.375885    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.375931    5282 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376166    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376183    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376188    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376207    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376215    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376223    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376230    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376235    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376239    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376244    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376248    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376252    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376257    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376261    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.391277 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376267    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376271    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376275    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376280    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376285    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376290    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376294    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376298    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376303    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376307    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376312    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376316    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376321    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376325    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376330    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376334    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376338    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376343    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376348    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.391787 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376352    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376357    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376362    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376369    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376376    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376381    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376386    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376390    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376394    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376399    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376403    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376408    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376412    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376417    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376421    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376425    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376430    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376435    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376439    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.392334 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376444    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376448    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376453    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376458    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376462    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376466    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376471    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376475    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376480    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376484    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376489    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376493    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376497    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376502    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376506    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376510    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376515    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376519    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376524    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376529    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.392812 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376534    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376538    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376543    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376547    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376551    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376556    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376560    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376564    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376569    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376574    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376578    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376583    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376587    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376591    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376596    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376600    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376604    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376609    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376614    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376618    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.393308 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376622    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376627    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376631    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376635    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376640    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376644    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376648    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376653    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376657    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376661    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376666    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376670    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376675    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376679    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.376725    5282 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376782    5282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:40:07.393796 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376790    5282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376795    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376800    5282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376805    5282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376812    5282 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376818    5282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376822    5282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376827    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376832    5282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376837    5282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376841    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376846    5282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376850    5282 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376854    5282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376859    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376864    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376868    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376872    5282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376876    5282 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376880    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:40:07.394171 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376885    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376889    5282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376893    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376897    5282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376902    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376906    5282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376910    5282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376914    5282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376918    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376924    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376929    5282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376933    5282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376938    5282 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376942    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376947    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376951    5282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376955    5282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376959    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376963    5282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376968    5282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:40:07.394689 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376973    5282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376977    5282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376981    5282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376986    5282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376990    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376994    5282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.376998    5282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377002    5282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377007    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377011    5282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377017    5282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377024    5282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377029    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377033    5282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377038    5282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377042    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377050    5282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377054    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377059    5282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:40:07.395170 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377064    5282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377069    5282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377073    5282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377080    5282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377084    5282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377089    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377093    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377100    5282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377104    5282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377108    5282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377113    5282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377117    5282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377121    5282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377126    5282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377130    5282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377135    5282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377140    5282 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377145    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377149    5282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:40:07.395745 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377154    5282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377158    5282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377163    5282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377167    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377174    5282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377183    5282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377188    5282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377209    5282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377214    5282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377218    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377225    5282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377229    5282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377233    5282 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377237    5282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377242    5282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377246    5282 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377250    5282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377257    5282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377262    5282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:40:07.396261 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377266    5282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377272    5282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377277    5282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377281    5282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377286    5282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377290    5282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377294    5282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:07.377298    5282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.377403    5282 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.379016    5282 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.379092    5282 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.379376    5282 server.go:1014] "Starting client certificate rotation"
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.379505    5282 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.379558    5282 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:33:56 +0000 UTC" deadline="2028-05-27 15:42:26.78103586 +0000 UTC"
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.379596    5282 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13700h2m19.401441283s"
Nov 03 19:40:07.396732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.380151    5282 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:40:07.397103 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.382113    5282 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:40:07.397103 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.383508    5282 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:40:07.397103 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.385284    5282 log.go:25] "Validated CRI v1 image API"
Nov 03 19:40:07.397103 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.385994    5282 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:40:07.397103 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.387904    5282 fs.go:135] Filesystem UUIDs: map[54fb9646-e257-4ff7-a506-e4af69766d17:/dev/nvme0n1p4 587d2e0a-137f-4ab4-ab15-1322921743b1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 03 19:40:07.397365 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.387926    5282 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0135e72e24b7928925edb15d299588c77ffe5456071c1ecdcfeb8ffaf69da267/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0135e72e24b7928925edb15d299588c77ffe5456071c1ecdcfeb8ffaf69da267/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/094ea790cbff5d2352269947c0c100c2ba7258c7cbf5150a7bef9af999e30895/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/094ea790cbff5d2352269947c0c100c2ba7258c7cbf5150a7bef9af999e30895/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1446892fef638100aee1a3f8e99093270174937a14745471cab1a56244fed52c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1446892fef638100aee1a3f8e99093270174937a14745471cab1a56244fed52c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/179fe18ca01b9400756c7ca2d092ed51c0fb86ed9b338c9790d9f0abac7a8d96/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/179fe18ca01b9400756c7ca2d092ed51c0fb86ed9b338c9790d9f0abac7a8d96/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19af5aa458d02ed4a0ee30f80b45f9a57688cd33fb59feb49e740adcba0f6109/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19af5aa458d02ed4a0ee30f80b45f9a57688cd33fb59feb49e740adcba0f6109/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d7dd793fe1b15f60cfb0a39f1e8005c883f8cd7bed5a0fefb2294f779796263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d7dd793fe1b15f60cfb0a39f1e8005c883f8cd7bed5a0fefb2294f779796263/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/401f75d98667e3ca023238787e853a4d3b72e59ccb780851c0a38866535bfa49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/401f75d98667e3ca023238787e853a4d3b72e59ccb780851c0a38866535bfa49/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b00fb0b96fbf679b568c28aa798c4cd9c8113cfe144e0fcf1bdbd414bccab3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b00fb0b96fbf679b568c28aa798c4cd9c8113cfe144e0fcf1bdbd414bccab3a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b0306d8e3bbdeb835e588ceff000b76040076c0b1fbf4d7169182f51f39b3e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b0306d8e3bbdeb835e588ceff000b76040076c0b1fbf4d7169182f51f39b3e2/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fc07e2911af26dec136f6d483ac02be64dbc278231996d1046e713a2cc53712/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fc07e2911af26dec136f6d483ac02be64dbc278231996d1046e713a2cc53712/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/632b44e40acc360cc62dd8eb6572293dd105ce73443bca8909ffbb2ab0cb0dd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/632b44e40acc360cc62dd8eb6572293dd105ce73443bca8909ffbb2ab0cb0dd0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a4fe582af93d4a63d2d2e433745d7b3b1d58d0b5e916e4d8e4549166e33e735/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a4fe582af93d4a63d2d2e433745d7b3b1d58d0b5e916e4d8e4549166e33e735/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82b208109713187177ff5ad0ff5b90d3247f5592764737a15dae8db3c39950eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82b208109713187177ff5ad0ff5b90d3247f5592764737a15dae8db3c39950eb/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/863d28a47b9f6679bf41d3df660c67eba3b824c17d92effdc5159e7e738d2ea2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/863d28a47b9f6679bf41d3df660c67eba3b824c17d92effdc5159e7e738d2ea2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aadc46d634086940923128e7ab65c7eeeb939f7477497be0653d4fa63908cbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aadc46d634086940923128e7ab65c7eeeb939f7477497be0653d4fa63908cbe/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d69387f8beb09e445d317a17b6a9059df08bfa4a4f1180d1318029e8ffd0850/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d69387f8beb09e445d317a17b6a9059df08bfa4a4f1180d1318029e8ffd0850/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b8cb613554626339b48afa2136aff35fa3fc728e9ef6b1ec608e0f9f4e4c2c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b8cb613554626339b48afa2136aff35fa3fc728e9ef6b1ec608e0f9f4e4c2c1/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a65d7756f3ce5da24700580ef223a05adc701675a7463fa5fba1a9bccf530ce3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a65d7756f3ce5da24700580ef223a05adc701675a7463fa5fba1a9bccf530ce3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d43aa2153a3723f97993b80cee8c6b450a7a944e2c290c33ce32f9ff782cb146/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d43aa2153a3723f97993b80cee8c6b450a7a944e2c290c33ce32f9ff782cb146/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee/volumes/kubernetes.io~projected/kube-api-access-tzkk5:{mountpoint:/var/lib/kubelet/pods/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee/volumes/kubernetes.io~projected/kube-api-access-tzkk5 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~projected/kube-api-access-hwxrd:{mountpoint:/var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~projected/kube-api-access-hwxrd major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fc369cd-d32a-4cad-9da6-d5bfd899591a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0fc369cd-d32a-4cad-9da6-d5bfd899591a/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26f64000-db36-453a-a425-a19b06a36f30/volumes/kubernetes.io~projected/kube-api-access-vkbx9:{mountpoint:/var/lib/kubelet/pods/26f64000-db36-453a-a425-a19b06a36f30/volumes/kubernetes.io~projected/kube-api-access-vkbx9 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c2e6638-f5a0-4258-a866-43417ff0f0c4/volumes/kubernetes.io~projected/kube-api-access-d4p94:{mountpoint:/var/lib/kubelet/pods/2c2e6638-f5a0-4258-a866-43417ff0f0c4/volumes/kubernetes.io~projected/kube-api-access-d4p94 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43f9158b-9674-477d-a64f-f19fe00bed5b/volumes/kubernetes.io~projected/kube-api-access-r458l:{mountpoint:/var/lib/kubelet/pods/43f9158b-9674-477d-a64f-f19fe00bed5b/volumes/kubernetes.io~projected/kube-api-access-r458l major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43f9158b-9674-477d-a64f-f19fe00bed5b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/43f9158b-9674-477d-a64f-f19fe00bed5b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~projected/kube-api-access-757sg:{mountpoint:/var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~projected/kube-api-access-757sg major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volumes/kubernetes.io~projected/kube-api-access-qjr9h:{mountpoint:/var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volumes/kubernetes.io~projected/kube-api-access-qjr9h major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e236c2-f2bc-4adc-8daf-f160c4c882f1/volumes/kubernetes.io~projected/kube-api-access-fp96d:{mountpoint:/var/lib/kubelet/pods/58e236c2-f2bc-4adc-8daf-f160c4c882f1/volumes/kubernetes.io~projected/kube-api-access-fp96d major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58e236c2-f2bc-4adc-8daf-f160c4c882f1/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/58e236c2-f2bc-4adc-8daf-f160c4c882f1/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b3aec1c-dcd1-4b89-86cc-898550befdad/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6b3aec1c-dcd1-4b89-86cc-898550befdad/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bb8d09d-f69e-4920-a257-0c957bebd264/volumes/kubernetes.io~projected/kube-api-access-qm9f8:{mountpoint:/var/lib/kubelet/pods/6bb8d09d-f69e-4920-a257-0c957bebd264/volumes/kubernetes.io~projected/kube-api-access-qm9f8 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bb8d09d-f69e-4920-a257-0c957bebd264/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6bb8d09d-f69e-4920-a257-0c957bebd264/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~projected/kube-api-access-qspcq:{mountpoint:/var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~projected/kube-api-access-qspcq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~projected/kube-api-access-nq8gj:{mountpoint:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~projected/kube-api-access-nq8gj major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9af4d64-b784-4e51-aa42-038905055202/volumes/kubernetes.io~projected/kube-api-access-npjs2:{mountpoint:/var/lib/kubelet/pods/a9af4d64-b784-4e51-aa42-038905055202/volumes/kubernetes.io~projected/kube-api-access-npjs2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9af4d64-b784-4e51-aa42-038905055202/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a9af4d64-b784-4e51-aa42-038905055202/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41/volumes/kubernetes.io~projected/kube-api-access-ctnkn:{mountpoint:/var/lib/kubelet/pods/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41/volumes/kubernetes.io~projected/kube-api-access-ctnkn major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41/volumes/kubernetes.io~secret/metrics-certs major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc78c3f5-3003-468d-82b1-bf29d59e5ea2/volumes/kubernetes.io~projected/kube-api-access-qbknc:{mountpoint:/var/lib/kubelet/pods/bc78c3f5-3003-468d-82b1-bf29d59e5ea2/volumes/kubernetes.io~projected/kube-api-access-qbknc major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce3c36fc-764d-4dc1-a515-000a6407542a/volumes/kubernetes.io~projected/kube-api-access-zt6pl:{mountpoint:/var/lib/kubelet/pods/ce3c36fc-764d-4dc1-a515-000a6407542a/volumes/kubernetes.io~projected/kube-api-access-zt6pl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2/volumes/kubernetes.io~projected/kube-api-access-mzg8t:{mountpoint:/var/lib/kubelet/pods/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2/volumes/kubernetes.io~projected/kube-api-access-mzg8t major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5ef855a-3c98-47fd-8559-bb40ad4db198/volumes/kubernetes.io~projected/kube-api-access-6zbd5:{mountpoint:/var/lib/kubelet/pods/e5ef855a-3c98-47fd-8559-bb40ad4db198/volumes/kubernetes.io~projected/kube-api-access-6zbd5 major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4fbf68c78219d820420808ece3f493f1e2abdeefaed15c4e1e5f1f5570ca1234/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/194da1a59cf07190745e3055ba9be0e4baf049750e2e44e2e8a77174c46aeafb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/41b28d4e15fe0d028cb57e2f4355f919381dee96769b9edc2e9d018605a0c601/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/148d72a78de7fc58c602f113b4ba3247eb7fb4a62de49e925fff9465347f5947/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1fd6c4ac3dd03f17aa870f9cd906e0689e9c2be5c50e9a82619978532a170622/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c3fd939c76b8a950c88e5891002c0e419a723d03feef608da3a237836a124510/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/3dba951af78fae1af9719d49e219d482c20d68eae9884d51b2ae261de1449cae/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5d078cd8ca16b577e2e52307fb00c746347dc9f5d8b4a685ff78f56848460102/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d09595fb4c3579bd340d3bc718fb9029eecd2c7aba9b8d6f5702d99842e82aef/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/87cb90cba34463ff84bacceb7491a14e812f64abe1dadad1aa73569c01d9f4f1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/b50ae5a6f0b77d2c987de810b7d8afa0b60ce3f81ca3946dda692e0ef4080a65/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/994f0aab2ac9c08375cd9cd623e45a3c4490014c61e4e722e4fb7bc53036036a/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c7022cbcece3ac31e0338d3eaacee80438d268fd542e6260fcec0f04b623e4c1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3a9137c7a2f8f5932ca83b57c3c1d235c963cea49c9fb529d6fcc64d23834659/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ef66cf8748c4cafdcd3ccce91d1d0b760c1b522445a76e349c059a7741cbfcc5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cc6b9d5ad5920da1207b14427f9e950213e5c8bd08e3e909d8888b258353b27f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/69677b108895f2c6d175a91da1cffaa555df84c62b1d6c1a2cddf2a22766d997/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/af9440882b38f9d03034218720670debf95ef271864d7fd19b30a2763396d7a9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/134475a66722b88ea8ccd9820ea0a91346b429609ac8403f497691432f602fb7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f40c1cb147d7144ca12bb01b7638ccc7523b6bd949a3857af3add2aa9f634b00/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3ac30d744ae9305ce25e80b78724a40cfcb9cf9174eb124476574212f97625dd/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/bbb6cfe011ed6191d30bc9d084fc1c81586ae5d043eb6bd6de54d3877c7c989c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3b170838360c4269892b3ef8dcb5d1af93814b9b19c5d0570816cd18a6072179/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/4047a56c2975885c0bfff42cc03e7ac83cc3eff9d55e0690e495dc97007742ef/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/46f5ea3a5f2548c7f2d931b205395e409f31f1e6de707678087c24d9c8babbf0/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/72a90a8825d3ef2be4236de78643f265958f7c3f72ce93170c21b0fa307d93b3/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/15c6560a90330d87a613dc102abbfb07598f4dcdba1b8d83b21d4d0e411e2862/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/b140be03bb09ad45a4df4bb74c2acadcf9705603bf4c0194fd28e3efe6b105ba/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/8a52cd6e7de54b5bafb9c5dedd2ad401db008700bd865330627b47eb570207d9/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/1fb22f91ca95606914363010c571e455edf2d4da46a664c2568343f74888bda2/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/66500339eda364e1796286f491949b18a0d41deee93940a77f7ba8c341a2ea77/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/e70829dbeb2c9dd040768f3a7c9394e04e27d4dda10bcb7077ac0228648006c9/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/e9e7e55bf3596cbce5588731c6780a1ca5621c95e33b52320d0cd17282db82ca/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/4ddd66412e830f36c8c595990b0d0fb37dd0cb4a62502109030f92175303afd0/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/0f88cf2a384c0096b30aa40401a141a3808048fa54e3bad97ec6d0f8d6208685/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/57c835a602f65b7998dc666965aaf85f5e040bb5aa6efa6f414b9382e0955ea0/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/bb6e2b4b273652f2c88716a0b61582154bc5773f26a86bed2aea8cb167eac555/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/45b4dabb68ae685b958fb3c0b5a21f8e0c6a6cdfe8a136f91ef27989f34414a4/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/ed309eedd9ab4e93281540dab2f01c2489b74b4a6271a76d6ba3a0ea13cc64f8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/9c01b799e67e65e519244ed680f46083042b0fa052bd74f721e63ba076a50f8e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/b7da09d0aaaa449b7fe231c6427153c2097c6ea8047478eec2cb8876088209e8/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/040fec000caaf13a58e8cc3971587ea4b9a7a629f05afc6149cdef84061a27e3/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/3573f537cbba3178c2347e1c45832ddb1f0c915e8b8a27d1b0e82383fc02e28d/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/216f7ac20f4cc6fd423c236a0c8d00f1abfe060b71df1e31044e26eec4857b45/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/681e590e7504014b4de23a84d5cc5b9807f69beb519bb8bb7d1a71f9677b99e9/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/598c0894f832cd6cfb6790fd948de1ad50010490ec6b2c25d6abeea587c6cd1b/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/04a2bc869dcfff2d1d539bfb6892063a31777eb0a4576e1c04aeb55ed104bdbf/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/db9bc0c43e09d69efff5a45ef881799a92e2852eeb6b09386d60003c0b53842f/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/2ea22ce32f17913c52550ca314c46ac3d992c62ab5e7899cf576a7bb26e929e4/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/20ee98ecc59fedb4627a1acfc247375172316e4a58c4cc01ae4ce8a5f8083cc1/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/f0f924b8156e1b25106e2f2cc6baef35dde822732028ec8b9d0e087f40906e0b/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/aad7e5617e9de564e7ab8e787eaa88a1ead817dd3668ea7efffc574aa771ade8/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/75045eb4a95e3f26c3b6f0e641f37426b86ddeec0fb3c621bb45ac663c818356/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/97cb596cc17c371d8a82ffc03a6a0f4e1b66ce1ce0ccf27447d2bad2e8667242/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f0eefba3f23b2738ecc64ca462667bd5ca86be91335f408655bd76b80de0688f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c6ea6e98af2af9fc8b63f231d54d5cf9fc5a1cf73315c659274b8063d1263de7/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/cd3cfe46f6e19ab780817e9909cefc9828f8332e6e00bf386c119d729f98bb0a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b4ae0b54c77f803ba4869a5f2a48d58413c6a87a9761c5b750022b5638a393f3/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/1fd48eb8d38ef5462b0ef5cf454f675becff380af5e0104938fd9854b799d9ff/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e1336e4de07e1eeca9820ff769b624c715ff3469462ba3ed1492674c8e8a7c9b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e33d249a4c9099c916806feeafac488d847824399eefa6dcb70b201d00b855a8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4c910c24e30e3ac715ac99fc045833f9aa228b2aba8e437f9f08102fb52e8967/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.396512    5282 manager.go:217] Machine: {Timestamp:2026-11-03 19:40:07.395980029 +0000 UTC m=+0.065181500 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097715 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ebf778b1c67e43996e7c5488401b1 SystemUUID:ec2ebf77-8b1c-67e4-3996-e7c5488401b1 BootID:1681ef20-97c9-41d0-ab8e-03c129ec1930 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2/volumes/kubernetes.io~projected/kube-api-access-mzg8t DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/401f75d98667e3ca023238787e853a4d3b72e59ccb780851c0a38866535bfa49/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b3aec1c-dcd1-4b89-86cc-898550befdad/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d69387f8beb09e445d317a17b6a9059df08bfa4a4f1180d1318029e8ffd0850/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fc07e2911af26dec136f6d483ac02be64dbc278231996d1046e713a2cc53712/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a65d7756f3ce5da24700580ef223a05adc701675a7463fa5fba1a9bccf530ce3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/632b44e40acc360cc62dd8eb6572293dd105ce73443bca8909ffbb2ab0cb0dd0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0135e72e24b7928925edb15d299588c77ffe5456071c1ecdcfeb8ffaf69da267/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~projected/kube-api-access-nq8gj DeviceMajor:0 DeviceMinor:281 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9af4d64-b784-4e51-aa42-038905055202/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b8cb613554626339b48afa2136aff35fa3fc728e9ef6b1ec608e0f9f4e4c2c1/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fc369cd-d32a-4cad-9da6-d5bfd899591a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43f9158b-9674-477d-a64f-f19fe00bed5b/volumes/kubernetes.io~projected/kube-api-access-r458l DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:280 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volumes/kubernetes.io~projected/kube-api-access-qjr9h DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~projected/kube-api-access-hwxrd DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aadc46d634086940923128e7ab65c7eeeb939f7477497be0653d4fa63908cbe/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b00fb0b96fbf679b568c28aa798c4cd9c8113cfe144e0fcf1bdbd414bccab3a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6bb8d09d-f69e-4920-a257-0c957bebd264/volumes/kubernetes.io~projected/kube-api-access-qm9f8 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82b208109713187177ff5ad0ff5b90d3247f5592764737a15dae8db3c39950eb/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6bb8d09d-f69e-4920-a257-0c957bebd264/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~projected/kube-api-access-qspcq DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d7dd793fe1b15f60cfb0a39f1e8005c883f8cd7bed5a0fefb2294f779796263/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d43aa2153a3723f97993b80cee8c6b450a7a944e2c290c33ce32f9ff782cb146/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/48f77550-ff58-46e6-b7d8-6840e4afa52c/volumes/kubernetes.io~projected/kube-api-access-757sg DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2c2e6638-f5a0-4258-a866-43417ff0f0c4/volumes/kubernetes.io~projected/kube-api-access-d4p94 DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58e236c2-f2bc-4adc-8daf-f160c4c882f1/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/094ea790cbff5d2352269947c0c100c2ba7258c7cbf5150a7bef9af999e30895/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58e236c2-f2bc-4adc-8daf-f160c4c882f1/volumes/kubernetes.io~projected/kube-api-access-fp96d DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5ef855a-3c98-47fd-8559-bb40ad4db198/volumes/kubernetes.io~projected/kube-api-access-6zbd5 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/179fe18ca01b9400756c7ca2d092ed51c0fb86ed9b338c9790d9f0abac7a8d96/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b0306d8e3bbdeb835e588ceff000b76040076c0b1fbf4d7169182f51f39b3e2/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce3c36fc-764d-4dc1-a515-000a6407542a/volumes/kubernetes.io~projected/kube-api-access-zt6pl DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/863d28a47b9f6679bf41d3df660c67eba3b824c17d92effdc5159e7e738d2ea2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19af5aa458d02ed4a0ee30f80b45f9a57688cd33fb59feb49e740adcba0f6109/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1446892fef638100aee1a3f8e99093270174937a14745471cab1a56244fed52c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9af4d64-b784-4e51-aa42-038905055202/volumes/kubernetes.io~projected/kube-api-access-npjs2 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee/volumes/kubernetes.io~projected/kube-api-access-tzkk5 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a4fe582af93d4a63d2d2e433745d7b3b1d58d0b5e916e4d8e4549166e33e735/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc78c3f5-3003-468d-82b1-bf29d59e5ea2/volumes/kubernetes.io~projected/kube-api-access-qbknc DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/076c5bac-22d5-4145-9dd6-bfc03e1b858e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41/volumes/kubernetes.io~projected/kube-api-access-ctnkn DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/26f64000-db36-453a-a425-a19b06a36f30/volumes/kubernetes.io~projected/kube-api-access-vkbx9 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/7b370ef7-e181-4b13-ba85-48090536f25c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/43f9158b-9674-477d-a64f-f19fe00bed5b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a113235-00b3-42db-ac16-0fc01da5a788/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0135e72e24b7928 MacAddress:4e:5b:3a:88:a9:fd Speed:10000 Mtu:8901} {Name:094ea790cbff5d2 MacAddress:c6:ed:29:56:b4:dd Speed:10000 Mtu:8901} {Name:19af5aa458d02ed MacAddress:4a:fd:21:91:59:01 Speed:10000 Mtu:8901} {Name:401f75d98667e3c MacAddress:f2:ee:46:6b:87:e1 Speed:10000 Mtu:8901} {Name:4fc07e2911af26d MacAddress:3e:1a:d4:22:19:d2 Speed:10000 Mtu:8901} {Name:6a4fe582af93d4a MacAddress:f2:12:c9:dc:46:9c Speed:10000 Mtu:8901} {Name:8d69387f8beb09e MacAddress:f2:a1:93:2f:1c:97 Speed:10000 Mtu:8901} {Name:9b8cb6135546263 MacAddress:a6:bd:d0:16:2c:4a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cb:76:30:ff Speed:0 Mtu:9001} {Name:br-int MacAddress:16:2c:de:d9:38:a5 Speed:0 Mtu:8901} {Name:d43aa2153a3723f MacAddress:9a:97:14:74:78:88 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cb:76:30:ff Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:10:46:9c:f5:bb Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:b9:88:b9:1d:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.396868    5282 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.396916    5282 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397226    5282 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397252    5282 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-244.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397384    5282 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397391    5282 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397402    5282 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397415    5282 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:40:07.397582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397522    5282 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:40:07.397980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397693    5282 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:40:07.397980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397809    5282 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:40:07.397980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397846    5282 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:40:07.397980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397859    5282 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:40:07.397980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397869    5282 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:40:07.397980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397880    5282 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:40:07.397980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.397889    5282 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:40:07.398867 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.398850    5282 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:40:07.398908 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.398869    5282 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:40:07.399490 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.399472    5282 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:40:07.402350 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402334    5282 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:40:07.402350 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402349    5282 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:40:07.402471 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402461    5282 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 19:40:07.402679 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402667    5282 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:40:07.402709 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402697    5282 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:40:07.402807 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402798    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:40:07.402835 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402812    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:40:07.402835 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402818    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:40:07.402835 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402825    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:40:07.402835 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402831    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:40:07.402940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402839    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:40:07.402940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402848    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:40:07.402940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402858    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:40:07.402940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402868    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:40:07.402940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402876    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:40:07.402940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402893    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:40:07.402940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402904    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:40:07.403119 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402982    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:40:07.403119 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.402993    5282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:40:07.406963 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.406949    5282 server.go:1263] "Started kubelet"
Nov 03 19:40:07.407294 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.407073    5282 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:40:07.408360 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.407284    5282 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:40:07.408305 ip-10-0-142-244 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:40:07.408548 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.407334    5282 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:40:07.409059 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.408948    5282 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:40:07.411036 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.410989    5282 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.413209    5282 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.413253    5282 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.413312    5282 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:33:57 +0000 UTC" deadline="2028-04-18 18:22:26.295628361 +0000 UTC"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.413352    5282 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12766h42m18.882279146s"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:07.413405    5282 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-244.us-east-2.compute.internal\" not found"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.413427    5282 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.413447    5282 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:40:07.414529 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.413436    5282 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:40:07.415326 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415282    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26f64000-db36-453a-a425-a19b06a36f30" volumeName="kubernetes.io/projected/26f64000-db36-453a-a425-a19b06a36f30-kube-api-access-vkbx9" seLinuxMountContext=""
Nov 03 19:40:07.415419 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415329    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 19:40:07.415419 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415345    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 19:40:07.415419 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415360    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 19:40:07.415419 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415371    5282 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:07.415620 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415374    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c2e6638-f5a0-4258-a866-43417ff0f0c4" volumeName="kubernetes.io/configmap/2c2e6638-f5a0-4258-a866-43417ff0f0c4-iptables-alerter-script" seLinuxMountContext=""
Nov 03 19:40:07.415620 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415599    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f9158b-9674-477d-a64f-f19fe00bed5b" volumeName="kubernetes.io/secret/43f9158b-9674-477d-a64f-f19fe00bed5b-metrics-tls" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415624    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f77550-ff58-46e6-b7d8-6840e4afa52c" volumeName="kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415638    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bb8d09d-f69e-4920-a257-0c957bebd264" volumeName="kubernetes.io/secret/6bb8d09d-f69e-4920-a257-0c957bebd264-cert" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415655    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/configmap/9a113235-00b3-42db-ac16-0fc01da5a788-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415669    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9af4d64-b784-4e51-aa42-038905055202" volumeName="kubernetes.io/projected/a9af4d64-b784-4e51-aa42-038905055202-kube-api-access-npjs2" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415684    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f9158b-9674-477d-a64f-f19fe00bed5b" volumeName="kubernetes.io/empty-dir/43f9158b-9674-477d-a64f-f19fe00bed5b-tmp-dir" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415698    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b370ef7-e181-4b13-ba85-48090536f25c" volumeName="kubernetes.io/projected/7b370ef7-e181-4b13-ba85-48090536f25c-kube-api-access-qspcq" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415711    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ef855a-3c98-47fd-8559-bb40ad4db198" volumeName="kubernetes.io/projected/e5ef855a-3c98-47fd-8559-bb40ad4db198-kube-api-access-6zbd5" seLinuxMountContext=""
Nov 03 19:40:07.415724 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415724    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee" volumeName="kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415739    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="076c5bac-22d5-4145-9dd6-bfc03e1b858e" volumeName="kubernetes.io/projected/076c5bac-22d5-4145-9dd6-bfc03e1b858e-kube-api-access-hwxrd" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415753    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="076c5bac-22d5-4145-9dd6-bfc03e1b858e" volumeName="kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415779    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ec13f7e-eee1-4082-bc4d-d71e59ef78c1" volumeName="kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-config" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415795    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bb8d09d-f69e-4920-a257-0c957bebd264" volumeName="kubernetes.io/projected/6bb8d09d-f69e-4920-a257-0c957bebd264-kube-api-access-qm9f8" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415807    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce3c36fc-764d-4dc1-a515-000a6407542a" volumeName="kubernetes.io/projected/ce3c36fc-764d-4dc1-a515-000a6407542a-kube-api-access-zt6pl" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415820    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" volumeName="kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415836    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="076c5bac-22d5-4145-9dd6-bfc03e1b858e" volumeName="kubernetes.io/configmap/076c5bac-22d5-4145-9dd6-bfc03e1b858e-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415848    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f77550-ff58-46e6-b7d8-6840e4afa52c" volumeName="kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415860    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b370ef7-e181-4b13-ba85-48090536f25c" volumeName="kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-tmp" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415873    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415886    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" volumeName="kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415898    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" volumeName="kubernetes.io/projected/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-kube-api-access-mzg8t" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415910    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ec13f7e-eee1-4082-bc4d-d71e59ef78c1" volumeName="kubernetes.io/projected/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-kube-api-access-qjr9h" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415922    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ec13f7e-eee1-4082-bc4d-d71e59ef78c1" volumeName="kubernetes.io/secret/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415944    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41" volumeName="kubernetes.io/projected/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-kube-api-access-ctnkn" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415956    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41" volumeName="kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415968    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e236c2-f2bc-4adc-8daf-f160c4c882f1" volumeName="kubernetes.io/secret/58e236c2-f2bc-4adc-8daf-f160c4c882f1-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415980    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b3aec1c-dcd1-4b89-86cc-898550befdad" volumeName="kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416004    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9af4d64-b784-4e51-aa42-038905055202" volumeName="kubernetes.io/empty-dir/a9af4d64-b784-4e51-aa42-038905055202-data-volume" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416017    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f9158b-9674-477d-a64f-f19fe00bed5b" volumeName="kubernetes.io/configmap/43f9158b-9674-477d-a64f-f19fe00bed5b-config-volume" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416029    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43f9158b-9674-477d-a64f-f19fe00bed5b" volumeName="kubernetes.io/projected/43f9158b-9674-477d-a64f-f19fe00bed5b-kube-api-access-r458l" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416042    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f77550-ff58-46e6-b7d8-6840e4afa52c" volumeName="kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416054    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ec13f7e-eee1-4082-bc4d-d71e59ef78c1" volumeName="kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416067    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b370ef7-e181-4b13-ba85-48090536f25c" volumeName="kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-etc-tuned" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416080    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce3c36fc-764d-4dc1-a515-000a6407542a" volumeName="kubernetes.io/empty-dir/ce3c36fc-764d-4dc1-a515-000a6407542a-tmp-dir" seLinuxMountContext=""
Nov 03 19:40:07.416090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416092    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="076c5bac-22d5-4145-9dd6-bfc03e1b858e" volumeName="kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416104    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26f64000-db36-453a-a425-a19b06a36f30" volumeName="kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416117    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c2e6638-f5a0-4258-a866-43417ff0f0c4" volumeName="kubernetes.io/projected/2c2e6638-f5a0-4258-a866-43417ff0f0c4-kube-api-access-d4p94" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416129    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416141    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26f64000-db36-453a-a425-a19b06a36f30" volumeName="kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-multus-daemon-config" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416155    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f77550-ff58-46e6-b7d8-6840e4afa52c" volumeName="kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-tls" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416169    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fc369cd-d32a-4cad-9da6-d5bfd899591a" volumeName="kubernetes.io/secret/0fc369cd-d32a-4cad-9da6-d5bfd899591a-agent-certs" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416188    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9af4d64-b784-4e51-aa42-038905055202" volumeName="kubernetes.io/secret/a9af4d64-b784-4e51-aa42-038905055202-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416227    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc78c3f5-3003-468d-82b1-bf29d59e5ea2" volumeName="kubernetes.io/projected/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kube-api-access-qbknc" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416240    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" volumeName="kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416254    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f77550-ff58-46e6-b7d8-6840e4afa52c" volumeName="kubernetes.io/empty-dir/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-textfile" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416266    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ec13f7e-eee1-4082-bc4d-d71e59ef78c1" volumeName="kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-env-overrides" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416279    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58e236c2-f2bc-4adc-8daf-f160c4c882f1" volumeName="kubernetes.io/projected/58e236c2-f2bc-4adc-8daf-f160c4c882f1-kube-api-access-fp96d" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416292    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/projected/9a113235-00b3-42db-ac16-0fc01da5a788-kube-api-access-nq8gj" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416306    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9af4d64-b784-4e51-aa42-038905055202" volumeName="kubernetes.io/configmap/a9af4d64-b784-4e51-aa42-038905055202-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416318    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5ef855a-3c98-47fd-8559-bb40ad4db198" volumeName="kubernetes.io/configmap/e5ef855a-3c98-47fd-8559-bb40ad4db198-serviceca" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416331    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48f77550-ff58-46e6-b7d8-6840e4afa52c" volumeName="kubernetes.io/projected/48f77550-ff58-46e6-b7d8-6840e4afa52c-kube-api-access-757sg" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416352    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/empty-dir/9a113235-00b3-42db-ac16-0fc01da5a788-tmp" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416366    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a113235-00b3-42db-ac16-0fc01da5a788" volumeName="kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-grpc-tls" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416379    5282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fc369cd-d32a-4cad-9da6-d5bfd899591a" volumeName="kubernetes.io/configmap/0fc369cd-d32a-4cad-9da6-d5bfd899591a-konnectivity-ca" seLinuxMountContext=""
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416391    5282 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416401    5282 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.415657    5282 factory.go:55] Registering systemd factory
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416504    5282 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416819    5282 factory.go:153] Registering CRI-O factory
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416834    5282 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416910    5282 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416932    5282 factory.go:103] Registering Raw factory
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.416946    5282 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:40:07.417443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.417295    5282 manager.go:319] Starting recovery of all containers
Nov 03 19:40:07.418765 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.417962    5282 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:07.418765 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.418273    5282 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:07.419488 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.419469    5282 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 19:40:07.432959 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.432916    5282 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:40:07.446891 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.446860    5282 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:40:07.446891 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.446888    5282 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:40:07.447083 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.446911    5282 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:40:07.447083 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:07.446953    5282 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:40:07.451655 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.451597    5282 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:40:07.458796 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.458762    5282 generic.go:356] "Generic (PLEG): container finished" podID="48f77550-ff58-46e6-b7d8-6840e4afa52c" containerID="59ffad534822c625910e487cfac976badf8497e1bf87423e17deb7a826d05d81" exitCode=0
Nov 03 19:40:07.475628 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.475598    5282 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="72b4d4fd4f8aebe8647c477f39952248265ef537a0826f13362b0c2d85443523" exitCode=0
Nov 03 19:40:07.475628 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.475620    5282 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="c3c650924a31d9b8fc0bcda12197e533943972dce59b773d6c6175d741ae1f33" exitCode=0
Nov 03 19:40:07.475628 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.475626    5282 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="e35995db7a430e9d68d8a826a7947256e2713d144fd788668ab86930318303a1" exitCode=0
Nov 03 19:40:07.475628 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.475634    5282 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="04e250a911388ad4a40dafc66fae03281b6bd332aa8ce87417d9f6ce6ba28e46" exitCode=0
Nov 03 19:40:07.475628 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.475639    5282 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="1a37025da157ac12d55eb37facd740b509af8f5484f75b8bcd85c44d5aacd4c8" exitCode=0
Nov 03 19:40:07.475908 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.475645    5282 generic.go:356] "Generic (PLEG): container finished" podID="cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2" containerID="27a7b090c3c93724f6432077c078320d8a74bc531fd61fbb91daab65cb5f0eaf" exitCode=0
Nov 03 19:40:07.490799 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.490761    5282 generic.go:356] "Generic (PLEG): container finished" podID="716ea44535b80bd6ca43cdb7c07af7ab" containerID="21741986a3156f047d0cb7e458a14048acd374927ca02526a13b5640c42aa1b1" exitCode=0
Nov 03 19:40:07.509493 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.509466    5282 manager.go:324] Recovery completed
Nov 03 19:40:07.528613 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528524    5282 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:40:07.528613 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528543    5282 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:40:07.528613 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528567    5282 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:40:07.528812 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528729    5282 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 19:40:07.528812 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528742    5282 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 19:40:07.528812 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528766    5282 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 19:40:07.528812 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528772    5282 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 19:40:07.528812 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528778    5282 policy_none.go:50] "Start"
Nov 03 19:40:07.528812 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528787    5282 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:40:07.528812 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528796    5282 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:40:07.529104 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528943    5282 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 19:40:07.529104 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528951    5282 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 19:40:07.529104 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.528958    5282 policy_none.go:44] "Start"
Nov 03 19:40:07.533607 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.533577    5282 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:40:07.536895 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.536880    5282 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:40:07.537751 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:07.536909    5282 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:40:07.537834 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.537761    5282 server.go:87] "Starting device plugin registration server"
Nov 03 19:40:07.538247 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.538228    5282 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:40:07.538318 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.538251    5282 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:40:07.538552 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.538533    5282 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:40:07.538675 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.538639    5282 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:40:07.538731 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.538692    5282 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:40:07.538731 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.538700    5282 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:40:07.538871 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.538746    5282 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:40:07.538674302Z","UUID":"e789aa0d-7522-4564-9520-e4f969f901f4","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:40:07.540930 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:07.540897    5282 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:40:07.542091 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.542068    5282 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:40:07.542227 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.542102    5282 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:40:07.547064 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.547021    5282 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"]
Nov 03 19:40:07.547735 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.547710    5282 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.547828 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.547758    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"716ea44535b80bd6ca43cdb7c07af7ab","Type":"ContainerStarted","Data":"d605ffb34fdc1288607284d50c2a48ba03b2ca6dc773fbefe64a9fe6d039d427"}
Nov 03 19:40:07.547828 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.547801    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"716ea44535b80bd6ca43cdb7c07af7ab","Type":"ContainerDied","Data":"21741986a3156f047d0cb7e458a14048acd374927ca02526a13b5640c42aa1b1"}
Nov 03 19:40:07.547828 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.547823    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal" event={"ID":"716ea44535b80bd6ca43cdb7c07af7ab","Type":"ContainerStarted","Data":"07d70229601b8b541f3cf8f48d8778225907c6c1bad16d8a2fddb9a1ff094c66"}
Nov 03 19:40:07.547959 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.547944    5282 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.563785 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.563752    5282 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:40:07.563911 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:07.563813    5282 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.564837 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.564815    5282 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:40:07.564886 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:07.564869    5282 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.620321 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.620285    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.620321 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.620321    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548ab8487a64671c8f29d89cabd4d700-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"548ab8487a64671c8f29d89cabd4d700\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.620523 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.620340    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.639406 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.639379    5282 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:40:07.641538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.641521    5282 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:40:07.641603 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.641551    5282 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:40:07.641603 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.641561    5282 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-244.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:40:07.641603 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.641602    5282 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.655006 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.654981    5282 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.655148 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.655057    5282 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.720834 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.720782    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.720834 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.720837    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.721059 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.720859    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548ab8487a64671c8f29d89cabd4d700-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"548ab8487a64671c8f29d89cabd4d700\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.721059 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.720913    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.721059 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.720941    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548ab8487a64671c8f29d89cabd4d700-config\") pod \"kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"548ab8487a64671c8f29d89cabd4d700\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:07.721059 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:07.720913    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/716ea44535b80bd6ca43cdb7c07af7ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" (UID: \"716ea44535b80bd6ca43cdb7c07af7ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:08.398265 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.398231    5282 apiserver.go:52] "Watching apiserver"
Nov 03 19:40:08.406839 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.406802    5282 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:40:08.407441 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.407406    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-79zbh","kube-system/konnectivity-agent-dlgxk","openshift-monitoring/node-exporter-b2qlc","openshift-multus/multus-additional-cni-plugins-xb2qp","kube-system/global-pull-secret-syncer-2g52t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal","openshift-multus/multus-dp7w8","openshift-network-operator/iptables-alerter-cncml","openshift-image-registry/node-ca-km679","kube-system/kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-t75cv","openshift-insights/insights-runtime-extractor-bv4d2","openshift-ovn-kubernetes/ovnkube-node-w5md5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t","openshift-dns/dns-default-8fqbg","openshift-ingress-canary/ingress-canary-st6kb","openshift-multus/network-metrics-daemon-bxkbj","openshift-monitoring/thanos-querier-5c8cd547dc-9csvs","openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5","openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h","openshift-dns/node-resolver-cjqb6"]
Nov 03 19:40:08.418577 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.418538    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:40:08.418889 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.418664    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:40:08.418889 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.418552    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:40:08.418889 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.418723    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.418889 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.418783    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:40:08.418889 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.418888    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.419218 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.418786    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.419218 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419060    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:40:08.419218 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419095    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.419218 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419114    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:40:08.419393 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419324    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.419460 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419438    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:40:08.419604 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419586    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.419658 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419620    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:40:08.419734 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419715    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.419866 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419846    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.419920 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419867    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.419920 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419890    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420015 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.419953    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.420066 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420032    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:40:08.420066 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420039    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.420155 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420082    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.420155 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420095    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420160    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420281 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420266    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420352 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420282    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.420352 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420288    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420352 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420334    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420406    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420427    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:40:08.420509 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420464    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:40:08.420565 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420550    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:40:08.420931 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420913    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:40:08.420994 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420946    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:40:08.420994 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.420947    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:40:08.423653 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423633    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-socket-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.423741 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423661    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-k8s-cni-cncf-io\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.423741 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423676    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-bin\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.423741 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423693    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-lib-modules\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.423890 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423760    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-tmp\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.423890 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423801    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc369cd-d32a-4cad-9da6-d5bfd899591a-agent-certs\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:40:08.423890 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423860    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-device-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.423987 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423900    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-tmp\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.423987 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423901    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-system-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.424059 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.423985    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9af4d64-b784-4e51-aa42-038905055202-data-volume\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.424059 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424021    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9af4d64-b784-4e51-aa42-038905055202-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.424156 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424072    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c36fc-764d-4dc1-a515-000a6407542a-tmp-dir\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:40:08.424156 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424084    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9af4d64-b784-4e51-aa42-038905055202-data-volume\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.424156 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424104    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6pl\" (UniqueName: \"kubernetes.io/projected/ce3c36fc-764d-4dc1-a515-000a6407542a-kube-api-access-zt6pl\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:40:08.424156 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424132    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ef855a-3c98-47fd-8559-bb40ad4db198-host\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:40:08.424331 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424155    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-slash\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.424331 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424158    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c36fc-764d-4dc1-a515-000a6407542a-tmp-dir\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:40:08.424331 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424186    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-etc-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.424331 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424238    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.424331 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424290    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.424331 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424322    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-kubelet-config\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.424606 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424433    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-netd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.424606 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424483    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-kubelet\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.424606 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424510    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43f9158b-9674-477d-a64f-f19fe00bed5b-config-volume\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.424606 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424527    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.424606 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424553    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.424606 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424583    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-multus-daemon-config\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424617    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ef855a-3c98-47fd-8559-bb40ad4db198-serviceca\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424645    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c2e6638-f5a0-4258-a866-43417ff0f0c4-iptables-alerter-script\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424671    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-node-log\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424668    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43f9158b-9674-477d-a64f-f19fe00bed5b-config-volume\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424728    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424757    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-conf\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424780    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-run\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424790    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-multus-daemon-config\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424822    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-cnibin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424852    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c2e6638-f5a0-4258-a866-43417ff0f0c4-iptables-alerter-script\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:40:08.424857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424851    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-socket-dir-parent\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424887    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovn-node-metrics-cert\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424928    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bb8d09d-f69e-4920-a257-0c957bebd264-cert\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424955    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.424982    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/076c5bac-22d5-4145-9dd6-bfc03e1b858e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425009    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425036    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjs2\" (UniqueName: \"kubernetes.io/projected/a9af4d64-b784-4e51-aa42-038905055202-kube-api-access-npjs2\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425062    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425090    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cnibin\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425113    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysconfig\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425137    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-etc-tuned\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425172    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43f9158b-9674-477d-a64f-f19fe00bed5b-metrics-tls\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425235    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3c36fc-764d-4dc1-a515-000a6407542a-hosts-file\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425253    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b370ef7-e181-4b13-ba85-48090536f25c-etc-tuned\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425254    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/076c5bac-22d5-4145-9dd6-bfc03e1b858e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425301    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnkn\" (UniqueName: \"kubernetes.io/projected/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-kube-api-access-ctnkn\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425315    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5ef855a-3c98-47fd-8559-bb40ad4db198-serviceca\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425334    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-registration-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.425356 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425354    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425369    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-hostroot\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425399    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-config\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425422    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-env-overrides\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425444    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-netns\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425459    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-bin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425493    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbx9\" (UniqueName: \"kubernetes.io/projected/26f64000-db36-453a-a425-a19b06a36f30-kube-api-access-vkbx9\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425526    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-systemd\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425543    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425550    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-env-overrides\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425584    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9af4d64-b784-4e51-aa42-038905055202-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425600    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-config\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425608    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-netns\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425624    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-var-lib-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425682    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-script-lib\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425708    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjr9h\" (UniqueName: \"kubernetes.io/projected/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-kube-api-access-qjr9h\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425729    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9af4d64-b784-4e51-aa42-038905055202-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425733    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9f8\" (UniqueName: \"kubernetes.io/projected/6bb8d09d-f69e-4920-a257-0c957bebd264-kube-api-access-qm9f8\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425763    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-var-lib-kubelet\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425793    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425815    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425845    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-cni-binary-copy\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425870    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovnkube-script-lib\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425877    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c2e6638-f5a0-4258-a866-43417ff0f0c4-host-slash\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425911    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbd5\" (UniqueName: \"kubernetes.io/projected/e5ef855a-3c98-47fd-8559-bb40ad4db198-kube-api-access-6zbd5\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.425937    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-ovn\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426001    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426003    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-log-socket\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426038    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspcq\" (UniqueName: \"kubernetes.io/projected/7b370ef7-e181-4b13-ba85-48090536f25c-kube-api-access-qspcq\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426070    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.426081 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426084    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26f64000-db36-453a-a425-a19b06a36f30-cni-binary-copy\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426175    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-sys-fs\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426219    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-conf-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426240    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426252    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426255    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-kubernetes\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426310    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxrd\" (UniqueName: \"kubernetes.io/projected/076c5bac-22d5-4145-9dd6-bfc03e1b858e-kube-api-access-hwxrd\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426333    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc369cd-d32a-4cad-9da6-d5bfd899591a-konnectivity-ca\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426352    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426373    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f9158b-9674-477d-a64f-f19fe00bed5b-tmp-dir\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426400    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-dbus\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426424    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-systemd-units\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426448    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-modprobe-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426472    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-host\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426521    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0fc369cd-d32a-4cad-9da6-d5bfd899591a-konnectivity-ca\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426531    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f9158b-9674-477d-a64f-f19fe00bed5b-tmp-dir\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426531    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9af4d64-b784-4e51-aa42-038905055202-crio-socket\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426577    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzg8t\" (UniqueName: \"kubernetes.io/projected/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-kube-api-access-mzg8t\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426623    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r458l\" (UniqueName: \"kubernetes.io/projected/43f9158b-9674-477d-a64f-f19fe00bed5b-kube-api-access-r458l\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426644    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-multus\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426676    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-kubelet\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426702    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-multus-certs\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426735    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-etc-kubernetes\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426758    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426777    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426805    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426827    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4p94\" (UniqueName: \"kubernetes.io/projected/2c2e6638-f5a0-4258-a866-43417ff0f0c4-kube-api-access-d4p94\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426859    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-systemd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426930    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-sys\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.427233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.426986    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbknc\" (UniqueName: \"kubernetes.io/projected/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kube-api-access-qbknc\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.429689 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.429665    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 19:40:08.429926 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.429905    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 19:40:08.430011 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.429999    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rq9xl\""
Nov 03 19:40:08.430067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430008    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmwsk\""
Nov 03 19:40:08.430067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.429998    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:40:08.430067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430054    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbmvx\""
Nov 03 19:40:08.430162 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430085    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:40:08.430162 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430092    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d4wgq\""
Nov 03 19:40:08.430162 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430085    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rlc2f\""
Nov 03 19:40:08.430162 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430117    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-drbgz\""
Nov 03 19:40:08.430411 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430397    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zxqwg\""
Nov 03 19:40:08.430411 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430398    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:40:08.430488 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430398    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hnqg\""
Nov 03 19:40:08.430488 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.430445    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9dv56\""
Nov 03 19:40:08.431308 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431213    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 19:40:08.431308 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431229    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:40:08.431308 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431259    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-67k2x\""
Nov 03 19:40:08.431308 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431270    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzp2g\""
Nov 03 19:40:08.431308 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431275    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:40:08.431308 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431299    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:40:08.431308 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431306    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g7dvr\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431317    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431320    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4j9gs\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431450    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431504    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2rb8\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431516    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431559    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431579    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431619    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431629    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431643    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:40:08.431657 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431639    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 19:40:08.432001 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431677    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xvgk\""
Nov 03 19:40:08.432001 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431686    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t5rk8\""
Nov 03 19:40:08.432001 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431694    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tlzjx\""
Nov 03 19:40:08.432001 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431682    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d2i8be8p11s16\""
Nov 03 19:40:08.432001 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431678    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zcfm6\""
Nov 03 19:40:08.432001 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.431763    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mvxmh\""
Nov 03 19:40:08.435164 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.435142    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9af4d64-b784-4e51-aa42-038905055202-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.435321 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.435152    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0fc369cd-d32a-4cad-9da6-d5bfd899591a-agent-certs\") pod \"konnectivity-agent-dlgxk\" (UID: \"0fc369cd-d32a-4cad-9da6-d5bfd899591a\") " pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:40:08.435321 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.435240    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-ovn-node-metrics-cert\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.435321 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.435250    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.435321 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.435302    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bb8d09d-f69e-4920-a257-0c957bebd264-cert\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:40:08.435441 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.435405    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43f9158b-9674-477d-a64f-f19fe00bed5b-metrics-tls\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.436255 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.436238    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/076c5bac-22d5-4145-9dd6-bfc03e1b858e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.436365 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.436347    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b3aec1c-dcd1-4b89-86cc-898550befdad-original-pull-secret\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.436781 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.436759    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-metrics-certs\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:40:08.436972 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.436955    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzg8t\" (UniqueName: \"kubernetes.io/projected/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-kube-api-access-mzg8t\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.437103 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.437087    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r458l\" (UniqueName: \"kubernetes.io/projected/43f9158b-9674-477d-a64f-f19fe00bed5b-kube-api-access-r458l\") pod \"dns-default-8fqbg\" (UID: \"43f9158b-9674-477d-a64f-f19fe00bed5b\") " pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:08.437149 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.437113    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjs2\" (UniqueName: \"kubernetes.io/projected/a9af4d64-b784-4e51-aa42-038905055202-kube-api-access-npjs2\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.437149 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.437132    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjr9h\" (UniqueName: \"kubernetes.io/projected/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-kube-api-access-qjr9h\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.437277 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.437262    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbknc\" (UniqueName: \"kubernetes.io/projected/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kube-api-access-qbknc\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.437489 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.437470    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbx9\" (UniqueName: \"kubernetes.io/projected/26f64000-db36-453a-a425-a19b06a36f30-kube-api-access-vkbx9\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.437893 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.437872    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4p94\" (UniqueName: \"kubernetes.io/projected/2c2e6638-f5a0-4258-a866-43417ff0f0c4-kube-api-access-d4p94\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:40:08.438171 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.438155    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxrd\" (UniqueName: \"kubernetes.io/projected/076c5bac-22d5-4145-9dd6-bfc03e1b858e-kube-api-access-hwxrd\") pod \"prometheus-operator-9cd89b74c-ztz7h\" (UID: \"076c5bac-22d5-4145-9dd6-bfc03e1b858e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ztz7h"
Nov 03 19:40:08.438218 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.438156    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspcq\" (UniqueName: \"kubernetes.io/projected/7b370ef7-e181-4b13-ba85-48090536f25c-kube-api-access-qspcq\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.438384 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.438370    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnkn\" (UniqueName: \"kubernetes.io/projected/b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41-kube-api-access-ctnkn\") pod \"network-metrics-daemon-bxkbj\" (UID: \"b5ca8e71-a6ff-48a6-b58c-fdd2bbbc3c41\") " pod="openshift-multus/network-metrics-daemon-bxkbj"
Nov 03 19:40:08.438879 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.438865    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6pl\" (UniqueName: \"kubernetes.io/projected/ce3c36fc-764d-4dc1-a515-000a6407542a-kube-api-access-zt6pl\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:40:08.439233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.439216    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbd5\" (UniqueName: \"kubernetes.io/projected/e5ef855a-3c98-47fd-8559-bb40ad4db198-kube-api-access-6zbd5\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:40:08.439306 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.439291    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9f8\" (UniqueName: \"kubernetes.io/projected/6bb8d09d-f69e-4920-a257-0c957bebd264-kube-api-access-qm9f8\") pod \"ingress-canary-st6kb\" (UID: \"6bb8d09d-f69e-4920-a257-0c957bebd264\") " pod="openshift-ingress-canary/ingress-canary-st6kb"
Nov 03 19:40:08.441436 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.441406    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkk5\" (UniqueName: \"kubernetes.io/projected/00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee-kube-api-access-tzkk5\") pod \"network-check-target-79zbh\" (UID: \"00a10aa1-fa7a-4b73-9ffd-ca212b2b20ee\") " pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:08.501515 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.501487    5282 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:08.514192 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.514161    5282 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:40:08.515344 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.515326    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:40:08.516035 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.516020    5282 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:40:08.516089 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:08.516059    5282 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-244.us-east-2.compute.internal"
Nov 03 19:40:08.527907 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.527870    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.528067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.527916    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58e236c2-f2bc-4adc-8daf-f160c4c882f1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:08.528067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.527942    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-node-log\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.528067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.527966    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-conf\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.528067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.527991    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-socket-dir-parent\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528017    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cnibin\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.528067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528039    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-registration-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.528067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528066    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-hostroot\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528076    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-node-log\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528104    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-netns\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528162    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-cnibin\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528207    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528230    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-socket-dir-parent\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528188    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-conf\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528254    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/58e236c2-f2bc-4adc-8daf-f160c4c882f1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528247    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-registration-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528276    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-hostroot\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528319    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-bin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528327    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-netns\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528363    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-bin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528393    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-systemd\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528416    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-var-lib-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528433    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-ovn\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528452    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9af4d64-b784-4e51-aa42-038905055202-crio-socket\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528467    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-systemd\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.528482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528477    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-grpc-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528502    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-ovn\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528508    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-sys-fs\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528513    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9af4d64-b784-4e51-aa42-038905055202-crio-socket\") pod \"insights-runtime-extractor-bv4d2\" (UID: \"a9af4d64-b784-4e51-aa42-038905055202\") " pod="openshift-insights/insights-runtime-extractor-bv4d2"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528529    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-conf-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528471    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-var-lib-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528575    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528600    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-sys-fs\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528604    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-kubernetes\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528639    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528659    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-host\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528648    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-kubernetes\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528740    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-host\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528761    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-grpc-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528763    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528828    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757sg\" (UniqueName: \"kubernetes.io/projected/48f77550-ff58-46e6-b7d8-6840e4afa52c-kube-api-access-757sg\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528857    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-multus\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528883    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-kubelet\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528909    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-multus-certs\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528933    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-etc-kubernetes\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528956    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528980    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-sys\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529004    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-socket-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529015    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-tls\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529029    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-k8s-cni-cncf-io\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529052    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-lib-modules\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529070    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-etc-kubernetes\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529082    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-sys\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529108    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-cni-multus\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.528570    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-conf-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529114    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-wtmp\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529153    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-var-lib-kubelet\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529181    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-k8s-cni-cncf-io\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529226    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-host-run-multus-certs\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529264    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-sys\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529285    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-etc-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529311    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-lib-modules\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529319    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-etc-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.529343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529350    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp96d\" (UniqueName: \"kubernetes.io/projected/58e236c2-f2bc-4adc-8daf-f160c4c882f1-kube-api-access-fp96d\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529417    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-socket-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529485    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529550    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-kubelet-config\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.529514    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-kubelet-config\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530364    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-netd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530636    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530702    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-textfile\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530740    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-tls\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530802    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530821    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.530844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530841    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-run\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530867    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-netd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530915    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-cnibin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530956    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.530989    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysconfig\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531080    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-textfile\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531128    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3c36fc-764d-4dc1-a515-000a6407542a-hosts-file\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531148    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-run\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531159    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-tls\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531215    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-ovn-kubernetes\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531238    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531293    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-netns\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531296    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3c36fc-764d-4dc1-a515-000a6407542a-hosts-file\") pod \"node-resolver-cjqb6\" (UID: \"ce3c36fc-764d-4dc1-a515-000a6407542a\") " pod="openshift-dns/node-resolver-cjqb6"
Nov 03 19:40:08.531354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531349    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-cnibin\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531372    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531380    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-var-lib-kubelet\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531399    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2-system-cni-dir\") pod \"multus-additional-cni-plugins-xb2qp\" (UID: \"cfbc3cd0-c3c4-4355-aca3-fa215c9c8dc2\") " pod="openshift-multus/multus-additional-cni-plugins-xb2qp"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531349    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysconfig\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531422    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-run-netns\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531441    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-var-lib-kubelet\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531475    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c2e6638-f5a0-4258-a866-43417ff0f0c4-host-slash\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531505    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-log-socket\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531530    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531562    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531699    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531754    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-dbus\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531779    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-systemd-units\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531817    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-modprobe-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531851    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8gj\" (UniqueName: \"kubernetes.io/projected/9a113235-00b3-42db-ac16-0fc01da5a788-kube-api-access-nq8gj\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.531876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531882    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531913    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531938    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-systemd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.531970    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532000    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-metrics-client-ca\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532006    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532030    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-bin\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532061    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-device-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532069    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c2e6638-f5a0-4258-a866-43417ff0f0c4-host-slash\") pod \"iptables-alerter-cncml\" (UID: \"2c2e6638-f5a0-4258-a866-43417ff0f0c4\") " pod="openshift-network-operator/iptables-alerter-cncml"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532086    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-system-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532118    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-log-socket\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532119    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a113235-00b3-42db-ac16-0fc01da5a788-metrics-client-ca\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532165    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ef855a-3c98-47fd-8559-bb40ad4db198-host\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532233    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-slash\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532265    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532300    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532329    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-root\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532361    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a113235-00b3-42db-ac16-0fc01da5a788-tmp\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532378    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a113235-00b3-42db-ac16-0fc01da5a788-metrics-client-ca\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532393    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-kubelet\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532510    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5ef855a-3c98-47fd-8559-bb40ad4db198-host\") pod \"node-ca-km679\" (UID: \"e5ef855a-3c98-47fd-8559-bb40ad4db198\") " pod="openshift-image-registry/node-ca-km679"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532608    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-sysctl-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.532684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532675    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-multus-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532732    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-systemd\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532765    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-slash\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532810    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-cni-bin\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532826    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-run-openvswitch\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532872    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26f64000-db36-453a-a425-a19b06a36f30-system-cni-dir\") pod \"multus-dp7w8\" (UID: \"26f64000-db36-453a-a425-a19b06a36f30\") " pod="openshift-multus/multus-dp7w8"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532918    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc78c3f5-3003-468d-82b1-bf29d59e5ea2-device-dir\") pod \"aws-ebs-csi-driver-node-zjs5t\" (UID: \"bc78c3f5-3003-468d-82b1-bf29d59e5ea2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zjs5t"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.532971    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b3aec1c-dcd1-4b89-86cc-898550befdad-dbus\") pod \"global-pull-secret-syncer-2g52t\" (UID: \"6b3aec1c-dcd1-4b89-86cc-898550befdad\") " pod="kube-system/global-pull-secret-syncer-2g52t"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533032    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533076    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a113235-00b3-42db-ac16-0fc01da5a788-tmp\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533084    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-systemd-units\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533134    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ec13f7e-eee1-4082-bc4d-d71e59ef78c1-host-kubelet\") pod \"ovnkube-node-w5md5\" (UID: \"4ec13f7e-eee1-4082-bc4d-d71e59ef78c1\") " pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533292    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-accelerators-collector-config\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533410    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533472    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48f77550-ff58-46e6-b7d8-6840e4afa52c-metrics-client-ca\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.533744 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533574    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b370ef7-e181-4b13-ba85-48090536f25c-etc-modprobe-d\") pod \"tuned-t75cv\" (UID: \"7b370ef7-e181-4b13-ba85-48090536f25c\") " pod="openshift-cluster-node-tuning-operator/tuned-t75cv"
Nov 03 19:40:08.534428 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.533831    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a113235-00b3-42db-ac16-0fc01da5a788-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.535940 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.535912    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp96d\" (UniqueName: \"kubernetes.io/projected/58e236c2-f2bc-4adc-8daf-f160c4c882f1-kube-api-access-fp96d\") pod \"monitoring-plugin-77f4f76594-z2zd5\" (UID: \"58e236c2-f2bc-4adc-8daf-f160c4c882f1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:08.536496 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.536462    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-757sg\" (UniqueName: \"kubernetes.io/projected/48f77550-ff58-46e6-b7d8-6840e4afa52c-kube-api-access-757sg\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.538167 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.538143    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8gj\" (UniqueName: \"kubernetes.io/projected/9a113235-00b3-42db-ac16-0fc01da5a788-kube-api-access-nq8gj\") pod \"thanos-querier-5c8cd547dc-9csvs\" (UID: \"9a113235-00b3-42db-ac16-0fc01da5a788\") " pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:08.599577 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.599524    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:08.633757 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.633719    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-sys\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.633757 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.633758    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-wtmp\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.633955 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.633804    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-root\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.633955 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.633836    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-sys\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.633955 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.633858    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-root\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.633955 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.633923    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48f77550-ff58-46e6-b7d8-6840e4afa52c-node-exporter-wtmp\") pod \"node-exporter-b2qlc\" (UID: \"48f77550-ff58-46e6-b7d8-6840e4afa52c\") " pod="openshift-monitoring/node-exporter-b2qlc"
Nov 03 19:40:08.727906 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:08.727872    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:09.198497 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:09.198417    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:09.312533 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:09.312497    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:09.423065 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:09.423030    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:09.584608 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:09.584570    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8fqbg"
Nov 03 19:40:09.782841 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:09.782805    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:09.786372 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:09.786350    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-z2zd5"
Nov 03 19:40:10.267190 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:10.267154    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:10.409092 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:10.409059    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:10.727332 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:10.727301    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:10.797491 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:10.797467    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:10.832561 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:10.832528    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:12.532404 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:12.532373    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs"
Nov 03 19:40:16.016071 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:16.016031    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:16.179501 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:16.179468    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:17.148011 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:17.147976    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:17.292973 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:17.292941    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:18.086828 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:18.086799    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:18.087087 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:18.087070    5282 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:40:18.095750 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:18.095724    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:18.978934 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:18.978899    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:19.141879 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:19.141796    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:25.492918 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:25.492866    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:25.612532 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:25.612498    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:29.536464 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:29.536435    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dlgxk"
Nov 03 19:40:31.612805 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:40:31.612767    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:40:32.468073 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:32.468044    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w5md5"
Nov 03 19:40:37.592949 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:37.592914    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:37.807419 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:37.807386    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:38.098272 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:38.098245    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-79zbh"
Nov 03 19:40:42.637852 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.637818    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"]
Nov 03 19:40:42.674215 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.674176    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"]
Nov 03 19:40:42.674363 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.674312    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.679433 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.679384    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 19:40:42.682478 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.682455    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 19:40:42.719266 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.719238    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rns48\""
Nov 03 19:40:42.719266 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.719254    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 19:40:42.719450 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.719291    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 19:40:42.719450 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.719313    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 19:40:42.719450 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.719347    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 19:40:42.770556 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770521    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-telemeter-client-tls\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.770723 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770563    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.770723 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770628    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-federate-client-tls\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.770723 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770679    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-serving-certs-ca-bundle\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.770723 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770718    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-metrics-client-ca\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.770874 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770773    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjnqw\" (UniqueName: \"kubernetes.io/projected/dcbb4036-9624-4a3f-b866-af74b917bd07-kube-api-access-xjnqw\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.770874 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770792    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-secret-telemeter-client\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.770874 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.770813    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.871957 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.871918    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-federate-client-tls\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.871957 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.871958    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-serving-certs-ca-bundle\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.872269 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.871998    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-metrics-client-ca\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.872269 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.872048    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjnqw\" (UniqueName: \"kubernetes.io/projected/dcbb4036-9624-4a3f-b866-af74b917bd07-kube-api-access-xjnqw\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.872269 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.872073    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-secret-telemeter-client\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.872422 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.872277    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.872422 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.872338    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-telemeter-client-tls\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.872422 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.872378    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.872555 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.872449    5282 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:40:42.873139 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.873111    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-serving-certs-ca-bundle\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.873139 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.873136    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-metrics-client-ca\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.873343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.873291    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcbb4036-9624-4a3f-b866-af74b917bd07-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.875164 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.875117    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-telemeter-client-tls\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.875164 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.875155    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-federate-client-tls\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.875587 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.875566    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.875661 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.875626    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dcbb4036-9624-4a3f-b866-af74b917bd07-secret-telemeter-client\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.879378 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.879360    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjnqw\" (UniqueName: \"kubernetes.io/projected/dcbb4036-9624-4a3f-b866-af74b917bd07-kube-api-access-xjnqw\") pod \"telemeter-client-59cf774bd5-lp2nw\" (UID: \"dcbb4036-9624-4a3f-b866-af74b917bd07\") " pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:42.989601 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:42.989558    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"
Nov 03 19:40:43.271087 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:43.270953    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59cf774bd5-lp2nw"]
Nov 03 19:40:43.274241 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:40:43.274209    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbb4036_9624_4a3f_b866_af74b917bd07.slice/crio-33bc2b8391765028cce39daffbbeb7e5ea775c6abba4fb23607d2d8e5ed99d49 WatchSource:0}: Error finding container 33bc2b8391765028cce39daffbbeb7e5ea775c6abba4fb23607d2d8e5ed99d49: Status 404 returned error can't find the container with id 33bc2b8391765028cce39daffbbeb7e5ea775c6abba4fb23607d2d8e5ed99d49
Nov 03 19:40:43.278238 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:43.276890    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:40:43.613754 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:43.613677    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw" event={"ID":"dcbb4036-9624-4a3f-b866-af74b917bd07","Type":"ContainerStarted","Data":"33bc2b8391765028cce39daffbbeb7e5ea775c6abba4fb23607d2d8e5ed99d49"}
Nov 03 19:40:44.336833 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:44.336797    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:44.466908 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:44.466861    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:45.622011 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:45.621976    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw" event={"ID":"dcbb4036-9624-4a3f-b866-af74b917bd07","Type":"ContainerStarted","Data":"6a59352ee97a613c5d98ad7debdb4d78b9bc96821218cd7dbe9adcdf6325a528"}
Nov 03 19:40:46.627580 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:46.627473    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw" event={"ID":"dcbb4036-9624-4a3f-b866-af74b917bd07","Type":"ContainerStarted","Data":"1bd87ecbcd70d4f9c63662275ebea4d8c08315367c175d31fc0fffa01e5b3806"}
Nov 03 19:40:47.634443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:47.634407    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw" event={"ID":"dcbb4036-9624-4a3f-b866-af74b917bd07","Type":"ContainerStarted","Data":"f1583a36b289f186717c75aa834d0c7c89e1c2270733e9e0d5aafff649754b24"}
Nov 03 19:40:47.655807 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:47.655759    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-59cf774bd5-lp2nw" podStartSLOduration=2.445496442 podStartE2EDuration="5.655745267s" totalImagesPullingTime="3.210248825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:47.654510073 +0000 UTC m=+40.323711555" watchObservedRunningTime="2026-11-03 19:40:47.655745267 +0000 UTC m=+40.324946742"
Nov 03 19:40:47.850594 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:47.850556    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:47.958965 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:47.958865    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:48.211156 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:48.211053    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:48.328782 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:40:48.328750    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:01.612960 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:41:01.612917    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:41:02.820255 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:02.820213    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:02.934836 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:02.934804    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:08.649020 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:08.648964    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:08.768142 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:08.768102    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:08.981362 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:08.981336    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:15.477120 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:15.477071    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:15.591172 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:15.591131    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:18.976945 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:18.976884    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:31.051554 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:31.051510    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:31.188493 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:31.188462    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:31.612752 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:41:31.612715    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:41:41.993231 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:41:41.993183    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:01.612600 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:42:01.612514    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:42:07.409932 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:42:07.409892    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:07.535700 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:42:07.535672    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:12.582280 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:42:12.582242    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:12.686548 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:42:12.686514    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:19.049618 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:42:19.049582    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:31.612522 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:42:31.612485    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:42:40.189208 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:42:40.189156    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:40.300452 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:42:40.300417    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:01.612397 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:43:01.612355    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:43:14.677282 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:14.677246    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:16.177701 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:16.177666    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:19.063022 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:19.062989    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:31.611675 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:43:31.611642    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:43:40.375502 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:40.375458    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:40.603376 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:40.603346    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:41.909353 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:41.909313    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:42.027688 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:42.027652    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:43.334136 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:43.334102    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:43.445816 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:43:43.445779    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:01.614876 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:44:01.614818    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:44:03.855489 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:44:03.855442    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:19.117664 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:44:19.117627    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:31.613304 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:44:31.613266    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:44:56.653810 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:44:56.653716    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:01.612479 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:45:01.612442    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:45:07.403884 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:07.403830    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:07.561060 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:07.561017    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:08.281371 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.281337    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"]
Nov 03 19:45:08.284582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.284566    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.290352 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.290328    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hv5kv\""
Nov 03 19:45:08.301472 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.301447    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 19:45:08.301617 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.301352    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 19:45:08.356186 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.356154    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"]
Nov 03 19:45:08.380465 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.380437    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"]
Nov 03 19:45:08.383600 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.383583    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.391016 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.390997    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-wtc8j\""
Nov 03 19:45:08.394096 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.394074    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"]
Nov 03 19:45:08.433423 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.433388    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.433837 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.433434    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pq9\" (UniqueName: \"kubernetes.io/projected/3630626f-3964-4fa6-bc9c-4f96c21c8521-kube-api-access-h9pq9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.433837 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.433558    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.535003 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.534908    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.535003 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.534957    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.535003 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.534988    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pq9\" (UniqueName: \"kubernetes.io/projected/3630626f-3964-4fa6-bc9c-4f96c21c8521-kube-api-access-h9pq9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.535313 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.535028    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.535313 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.535051    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.535313 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.535177    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58l6t\" (UniqueName: \"kubernetes.io/projected/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-kube-api-access-58l6t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.535461 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.535351    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.535461 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.535400    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.543590 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.543566    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pq9\" (UniqueName: \"kubernetes.io/projected/3630626f-3964-4fa6-bc9c-4f96c21c8521-kube-api-access-h9pq9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.605010 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.604971    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:08.636090 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.636057    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.636236 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.636100    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58l6t\" (UniqueName: \"kubernetes.io/projected/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-kube-api-access-58l6t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.636486 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.636460    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.636637 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.636507    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.636773 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.636753    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.643967 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.643941    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58l6t\" (UniqueName: \"kubernetes.io/projected/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-kube-api-access-58l6t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.699568 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.699538    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:08.779035 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.779000    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"]
Nov 03 19:45:08.782265 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.782248    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.785694 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.785647    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:45:08.785775 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.785730    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:45:08.785775 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.785763    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:45:08.786032 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.786014    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gjlpz\""
Nov 03 19:45:08.786174 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.786162    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:45:08.786263 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.786252    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:45:08.786302 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.786273    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:45:08.794798 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.794777    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"]
Nov 03 19:45:08.840570 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.840538    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:08.872950 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.872921    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"]
Nov 03 19:45:08.875869 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:45:08.875849    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3630626f_3964_4fa6_bc9c_4f96c21c8521.slice/crio-dff2453c7c36e1cead84ed247c1e43335a74e7981bfac5ede6f7e9a2c06938fe WatchSource:0}: Error finding container dff2453c7c36e1cead84ed247c1e43335a74e7981bfac5ede6f7e9a2c06938fe: Status 404 returned error can't find the container with id dff2453c7c36e1cead84ed247c1e43335a74e7981bfac5ede6f7e9a2c06938fe
Nov 03 19:45:08.938976 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.938941    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.939128 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.938985    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.939128 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.939019    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1a074-4d72-4490-b193-a4e3e8003047-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.939128 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.939110    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.939333 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.939165    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.939333 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.939272    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.939399 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.939337    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lxn\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-kube-api-access-x7lxn\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:08.952096 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.952063    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:08.969122 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:08.969096    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"]
Nov 03 19:45:08.972564 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:45:08.972542    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8aca5d_0916_4fcc_aa9b_89f6c19f73b7.slice/crio-4b614599254d2d8c74e126db66c828dc24a738ed70b0de22136858a37169d3c2 WatchSource:0}: Error finding container 4b614599254d2d8c74e126db66c828dc24a738ed70b0de22136858a37169d3c2: Status 404 returned error can't find the container with id 4b614599254d2d8c74e126db66c828dc24a738ed70b0de22136858a37169d3c2
Nov 03 19:45:09.040501 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.040421    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lxn\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-kube-api-access-x7lxn\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.040501 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.040460    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.040501 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.040479    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.040770 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.040506    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1a074-4d72-4490-b193-a4e3e8003047-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.040770 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.040537    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.040770 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.040578    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.040770 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.040624    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.041138 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.041112    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.042929 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.042909    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.043019 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.042977    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.043180 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.043165    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1a074-4d72-4490-b193-a4e3e8003047-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.043358 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.043340    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.048002 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.047984    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.048269 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.048249    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lxn\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-kube-api-access-x7lxn\") pod \"istiod-openshift-gateway-7876bfcb74-b8qdl\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.093668 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.093639    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:09.365300 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.365212    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"]
Nov 03 19:45:09.368953 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:45:09.368927    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b1a074_4d72_4490_b193_a4e3e8003047.slice/crio-dcb7e427ee90ee937228be9cfe56c00f3dce3d3591c8b1fd8877e5bb3887fc36 WatchSource:0}: Error finding container dcb7e427ee90ee937228be9cfe56c00f3dce3d3591c8b1fd8877e5bb3887fc36: Status 404 returned error can't find the container with id dcb7e427ee90ee937228be9cfe56c00f3dce3d3591c8b1fd8877e5bb3887fc36
Nov 03 19:45:09.452727 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.452693    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz" event={"ID":"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7","Type":"ContainerStarted","Data":"4b614599254d2d8c74e126db66c828dc24a738ed70b0de22136858a37169d3c2"}
Nov 03 19:45:09.454255 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.454224    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl" event={"ID":"66b1a074-4d72-4490-b193-a4e3e8003047","Type":"ContainerStarted","Data":"dcb7e427ee90ee937228be9cfe56c00f3dce3d3591c8b1fd8877e5bb3887fc36"}
Nov 03 19:45:09.455448 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.455417    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp" event={"ID":"3630626f-3964-4fa6-bc9c-4f96c21c8521","Type":"ContainerStarted","Data":"dff2453c7c36e1cead84ed247c1e43335a74e7981bfac5ede6f7e9a2c06938fe"}
Nov 03 19:45:09.716922 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:09.716843    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:12.860040 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:12.860007    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:12.860418 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:12.860083    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:45:13.480708 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:13.480669    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl" event={"ID":"66b1a074-4d72-4490-b193-a4e3e8003047","Type":"ContainerStarted","Data":"d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa"}
Nov 03 19:45:13.481707 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:13.481673    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:13.484621 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:13.483362    5282 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-b8qdl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:45:13.484621 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:13.483423    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl" podUID="66b1a074-4d72-4490-b193-a4e3e8003047" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:45:13.513518 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:13.513315    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:13.627133 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:13.627095    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:14.310140 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:14.310106    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:14.441031 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:14.441000    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:14.486025 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:14.485997    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:45:14.507915 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:14.507860    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl" podStartSLOduration=3.020021204 podStartE2EDuration="6.507844553s" totalImagesPullingTime="3.487823349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:13.509658324 +0000 UTC m=+306.178859795" watchObservedRunningTime="2026-11-03 19:45:14.507844553 +0000 UTC m=+307.177046024"
Nov 03 19:45:15.489855 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:15.489823    5282 generic.go:356] "Generic (PLEG): container finished" podID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerID="353a8af2eed280da1d1d74b4ce7d0eac494929c27599f9429d66698d327e3bcd" exitCode=0
Nov 03 19:45:15.490307 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:15.489911    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp" event={"ID":"3630626f-3964-4fa6-bc9c-4f96c21c8521","Type":"ContainerDied","Data":"353a8af2eed280da1d1d74b4ce7d0eac494929c27599f9429d66698d327e3bcd"}
Nov 03 19:45:15.491576 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:15.491538    5282 generic.go:356] "Generic (PLEG): container finished" podID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerID="d8878c8fc9222d62f2cc98d7e0073619f063b808045f6156ae4b5a826bb23c88" exitCode=0
Nov 03 19:45:15.491689 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:15.491600    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz" event={"ID":"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7","Type":"ContainerDied","Data":"d8878c8fc9222d62f2cc98d7e0073619f063b808045f6156ae4b5a826bb23c88"}
Nov 03 19:45:16.501432 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:16.501393    5282 generic.go:356] "Generic (PLEG): container finished" podID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerID="4f8a6edfbe33f9b9b40b096cfacfbef2e456eeafe2656da66fa97a4d61b64674" exitCode=0
Nov 03 19:45:16.501854 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:16.501479    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz" event={"ID":"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7","Type":"ContainerDied","Data":"4f8a6edfbe33f9b9b40b096cfacfbef2e456eeafe2656da66fa97a4d61b64674"}
Nov 03 19:45:17.110911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:17.110871    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:18.512373 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:18.512338    5282 generic.go:356] "Generic (PLEG): container finished" podID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerID="5385b2cc82c79b6bd6008389c9169800163fc2fdfcf62b16971a0ab28bd16fa3" exitCode=0
Nov 03 19:45:18.512778 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:18.512418    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp" event={"ID":"3630626f-3964-4fa6-bc9c-4f96c21c8521","Type":"ContainerDied","Data":"5385b2cc82c79b6bd6008389c9169800163fc2fdfcf62b16971a0ab28bd16fa3"}
Nov 03 19:45:19.162212 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:19.162164    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:23.535364 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:23.535329    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp" event={"ID":"3630626f-3964-4fa6-bc9c-4f96c21c8521","Type":"ContainerStarted","Data":"d6d483ccd273e593436fea75fa5336745cd5ffdfa19102f71c779c63d80265f0"}
Nov 03 19:45:23.537333 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:23.537309    5282 generic.go:356] "Generic (PLEG): container finished" podID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerID="3ba44f2d4de8f2335dc8667ca1361167df8d0e4d8dbb3514f906a7fce3c3a11b" exitCode=0
Nov 03 19:45:23.537451 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:23.537352    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz" event={"ID":"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7","Type":"ContainerDied","Data":"3ba44f2d4de8f2335dc8667ca1361167df8d0e4d8dbb3514f906a7fce3c3a11b"}
Nov 03 19:45:23.552038 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:23.551994    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp" podStartSLOduration=1.7293574299999999 podStartE2EDuration="15.55198325s" totalImagesPullingTime="13.82262582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:23.550394561 +0000 UTC m=+316.219596030" watchObservedRunningTime="2026-11-03 19:45:23.55198325 +0000 UTC m=+316.221184717"
Nov 03 19:45:24.542355 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.542322    5282 generic.go:356] "Generic (PLEG): container finished" podID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerID="d6d483ccd273e593436fea75fa5336745cd5ffdfa19102f71c779c63d80265f0" exitCode=0
Nov 03 19:45:24.542736 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.542406    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp" event={"ID":"3630626f-3964-4fa6-bc9c-4f96c21c8521","Type":"ContainerDied","Data":"d6d483ccd273e593436fea75fa5336745cd5ffdfa19102f71c779c63d80265f0"}
Nov 03 19:45:24.663770 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.663746    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:24.753604 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.753573    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-kube-api-access-58l6t\" (UniqueName: \"kubernetes.io/projected/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-kube-api-access-58l6t\") pod \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") "
Nov 03 19:45:24.753759 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.753615    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-util\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-util\") pod \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") "
Nov 03 19:45:24.753759 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.753649    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-bundle\") pod \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\" (UID: \"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7\") "
Nov 03 19:45:24.754221 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.754169    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-bundle" pod "9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" (UID: "9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:24.755804 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.755778    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-kube-api-access-58l6t" pod "9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" (UID: "9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7"). InnerVolumeSpecName "kube-api-access-58l6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:24.757836 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.757815    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-util" pod "9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" (UID: "9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:24.854386 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.854308    5282 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-bundle\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:24.854386 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.854339    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58l6t\" (UniqueName: \"kubernetes.io/projected/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-kube-api-access-58l6t\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:24.854386 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:24.854348    5282 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7-util\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:25.547982 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.547955    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz"
Nov 03 19:45:25.548396 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.547976    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7179pfz" event={"ID":"9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7","Type":"ContainerDied","Data":"4b614599254d2d8c74e126db66c828dc24a738ed70b0de22136858a37169d3c2"}
Nov 03 19:45:25.548396 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.548025    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b614599254d2d8c74e126db66c828dc24a738ed70b0de22136858a37169d3c2"
Nov 03 19:45:25.663140 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.663117    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:25.760407 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.760374    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3630626f-3964-4fa6-bc9c-4f96c21c8521-kube-api-access-h9pq9\" (UniqueName: \"kubernetes.io/projected/3630626f-3964-4fa6-bc9c-4f96c21c8521-kube-api-access-h9pq9\") pod \"3630626f-3964-4fa6-bc9c-4f96c21c8521\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") "
Nov 03 19:45:25.760569 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.760415    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-util\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-util\") pod \"3630626f-3964-4fa6-bc9c-4f96c21c8521\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") "
Nov 03 19:45:25.760569 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.760469    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-bundle\") pod \"3630626f-3964-4fa6-bc9c-4f96c21c8521\" (UID: \"3630626f-3964-4fa6-bc9c-4f96c21c8521\") "
Nov 03 19:45:25.761138 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.761108    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-bundle" pod "3630626f-3964-4fa6-bc9c-4f96c21c8521" (UID: "3630626f-3964-4fa6-bc9c-4f96c21c8521"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:25.762449 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.762425    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3630626f-3964-4fa6-bc9c-4f96c21c8521-kube-api-access-h9pq9" pod "3630626f-3964-4fa6-bc9c-4f96c21c8521" (UID: "3630626f-3964-4fa6-bc9c-4f96c21c8521"). InnerVolumeSpecName "kube-api-access-h9pq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:25.767917 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.767893    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-util" pod "3630626f-3964-4fa6-bc9c-4f96c21c8521" (UID: "3630626f-3964-4fa6-bc9c-4f96c21c8521"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:25.861968 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.861928    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9pq9\" (UniqueName: \"kubernetes.io/projected/3630626f-3964-4fa6-bc9c-4f96c21c8521-kube-api-access-h9pq9\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:25.861968 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.861966    5282 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-util\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:25.862151 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:25.861983    5282 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3630626f-3964-4fa6-bc9c-4f96c21c8521-bundle\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:26.553191 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:26.553156    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp" event={"ID":"3630626f-3964-4fa6-bc9c-4f96c21c8521","Type":"ContainerDied","Data":"dff2453c7c36e1cead84ed247c1e43335a74e7981bfac5ede6f7e9a2c06938fe"}
Nov 03 19:45:26.553572 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:26.553226    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dff2453c7c36e1cead84ed247c1e43335a74e7981bfac5ede6f7e9a2c06938fe"
Nov 03 19:45:26.553572 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:26.553166    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmzmjp"
Nov 03 19:45:31.612556 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:45:31.612520    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:45:47.837267 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:47.837223    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:47.949342 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:47.949304    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.441918 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.441890    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.620451 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620417    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"]
Nov 03 19:45:48.620696 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620683    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerName="pull"
Nov 03 19:45:48.620740 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620695    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerName="pull"
Nov 03 19:45:48.620740 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620707    5282 cpu_manager.go:414] "containerMap: removing container" podUID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerName="extract"
Nov 03 19:45:48.620740 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620712    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerName="extract"
Nov 03 19:45:48.620740 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620721    5282 cpu_manager.go:414] "containerMap: removing container" podUID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerName="util"
Nov 03 19:45:48.620740 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620726    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerName="util"
Nov 03 19:45:48.620740 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620739    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerName="util"
Nov 03 19:45:48.620911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620744    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerName="util"
Nov 03 19:45:48.620911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620759    5282 cpu_manager.go:414] "containerMap: removing container" podUID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerName="pull"
Nov 03 19:45:48.620911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620764    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerName="pull"
Nov 03 19:45:48.620911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620769    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerName="extract"
Nov 03 19:45:48.620911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620774    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerName="extract"
Nov 03 19:45:48.620911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620815    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3630626f-3964-4fa6-bc9c-4f96c21c8521" containerName="extract"
Nov 03 19:45:48.620911 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.620823    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8aca5d-0916-4fcc-aa9b-89f6c19f73b7" containerName="extract"
Nov 03 19:45:48.629249 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.629222    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.654561 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.654531    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"]
Nov 03 19:45:48.697052 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.696968    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 19:45:48.697052 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.696977    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 19:45:48.697289 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.697241    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 19:45:48.697289 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.697256    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-85znx\""
Nov 03 19:45:48.697378 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.697298    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 19:45:48.697378 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.697350    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 19:45:48.722691 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.722654    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.722855 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.722698    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e296f90-4f55-42f9-9310-f31d1112881e-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.722855 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.722725    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqdz\" (UniqueName: \"kubernetes.io/projected/7e296f90-4f55-42f9-9310-f31d1112881e-kube-api-access-hbqdz\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.722950 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.722854    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.722950 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.722887    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e296f90-4f55-42f9-9310-f31d1112881e-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.722950 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.722929    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.787951 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.787916    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.824105 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.824075    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.824113    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e296f90-4f55-42f9-9310-f31d1112881e-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.824162    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.824190    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.824253    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e296f90-4f55-42f9-9310-f31d1112881e-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.824478 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.824432    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqdz\" (UniqueName: \"kubernetes.io/projected/7e296f90-4f55-42f9-9310-f31d1112881e-kube-api-access-hbqdz\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.824672 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.824628    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7e296f90-4f55-42f9-9310-f31d1112881e-manifests\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.826523 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.826496    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e296f90-4f55-42f9-9310-f31d1112881e-tmp\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.826716 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.826695    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.826938 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.826920    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.827111 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.827090    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e296f90-4f55-42f9-9310-f31d1112881e-metrics-cert\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.832506 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.832481    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqdz\" (UniqueName: \"kubernetes.io/projected/7e296f90-4f55-42f9-9310-f31d1112881e-kube-api-access-hbqdz\") pod \"opendatahub-operator-controller-manager-5749d45fbb-kqx6s\" (UID: \"7e296f90-4f55-42f9-9310-f31d1112881e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:48.893598 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.893569    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.942738 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:48.942709    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:49.024835 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:49.024810    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:49.227552 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:49.227522    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"]
Nov 03 19:45:49.231030 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:45:49.231004    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e296f90_4f55_42f9_9310_f31d1112881e.slice/crio-2edb33253392a653a21a30fe3f195b99d3c5bdc94741901e40bd70a0656a29f0 WatchSource:0}: Error finding container 2edb33253392a653a21a30fe3f195b99d3c5bdc94741901e40bd70a0656a29f0: Status 404 returned error can't find the container with id 2edb33253392a653a21a30fe3f195b99d3c5bdc94741901e40bd70a0656a29f0
Nov 03 19:45:49.232689 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:49.232668    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:45:49.632869 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:49.632787    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s" event={"ID":"7e296f90-4f55-42f9-9310-f31d1112881e","Type":"ContainerStarted","Data":"2edb33253392a653a21a30fe3f195b99d3c5bdc94741901e40bd70a0656a29f0"}
Nov 03 19:45:49.639901 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:49.639873    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:51.822727 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:51.822689    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:51.944145 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:51.944109    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:53.651799 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:53.651770    5282 generic.go:356] "Generic (PLEG): container finished" podID="7e296f90-4f55-42f9-9310-f31d1112881e" containerID="1ce3d628c303dbe41db748ef0885739215694bb6e9062eef05e4e49eff9fe286" exitCode=0
Nov 03 19:45:53.652145 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:53.651831    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s" event={"ID":"7e296f90-4f55-42f9-9310-f31d1112881e","Type":"ContainerDied","Data":"1ce3d628c303dbe41db748ef0885739215694bb6e9062eef05e4e49eff9fe286"}
Nov 03 19:45:54.009743 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:54.009700    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:54.656932 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:54.656895    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s" event={"ID":"7e296f90-4f55-42f9-9310-f31d1112881e","Type":"ContainerStarted","Data":"97d9963153044974d9f7a7899984a79d3231f003e9fc24a35472403bc1674849"}
Nov 03 19:45:54.657387 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:54.657009    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:45:54.688383 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:54.688327    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s" podStartSLOduration=1.9772742220000001 podStartE2EDuration="6.688308273s" totalImagesPullingTime="4.23220833s" totalInitContainerRuntime="478.825721ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:54.686450635 +0000 UTC m=+347.355652143" watchObservedRunningTime="2026-11-03 19:45:54.688308273 +0000 UTC m=+347.357509745"
Nov 03 19:45:55.267470 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:55.267425    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:56.799583 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:45:56.799538    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:01.601548 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:01.601513    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:01.612469 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:46:01.612437    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:46:02.842971 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:02.842924    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:03.807758 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:03.807710    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:03.927268 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:03.927228    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:04.800993 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:04.800954    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:05.662857 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:05.662826    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5749d45fbb-kqx6s"
Nov 03 19:46:05.665679 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:05.665649    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:05.813153 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:05.813119    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:06.340014 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:06.339973    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:06.557396 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:06.557363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:06.752293 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:06.752262    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:06.996269 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:06.996217    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:08.266787 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:08.266756    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:08.693952 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:08.693872    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:08.913440 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:08.913405    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:12.048663 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:12.048626    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:12.242390 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:12.242355    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:13.039702 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:13.039668    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.872076 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:15.872043    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:16.424480 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:16.424439    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:18.172607 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.172577    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"]
Nov 03 19:46:18.179080 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.179061    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.182521 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.182498    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"]
Nov 03 19:46:18.182826 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.182809    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 19:46:18.182947 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.182925    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 19:46:18.183259 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.183028    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 19:46:18.183259 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.183030    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 19:46:18.183259 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.183247    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lpjdc\""
Nov 03 19:46:18.219757 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.219728    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:18.330477 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.330446    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:18.352553 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.352523    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hq7j\" (UniqueName: \"kubernetes.io/projected/6beed4ed-3890-4845-952a-71bb8ed502bf-kube-api-access-9hq7j\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.352693 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.352569    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6beed4ed-3890-4845-952a-71bb8ed502bf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.352693 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.352587    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6beed4ed-3890-4845-952a-71bb8ed502bf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.453078 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.453002    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hq7j\" (UniqueName: \"kubernetes.io/projected/6beed4ed-3890-4845-952a-71bb8ed502bf-kube-api-access-9hq7j\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.453078 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.453050    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6beed4ed-3890-4845-952a-71bb8ed502bf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.453273 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.453079    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6beed4ed-3890-4845-952a-71bb8ed502bf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.453797 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.453773    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6beed4ed-3890-4845-952a-71bb8ed502bf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.455398 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.455380    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6beed4ed-3890-4845-952a-71bb8ed502bf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.460780 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.460752    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hq7j\" (UniqueName: \"kubernetes.io/projected/6beed4ed-3890-4845-952a-71bb8ed502bf-kube-api-access-9hq7j\") pod \"kuadrant-console-plugin-66c4fb575d-fbz8b\" (UID: \"6beed4ed-3890-4845-952a-71bb8ed502bf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.500995 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.500973    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"
Nov 03 19:46:18.767574 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.767545    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b"]
Nov 03 19:46:18.770643 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:46:18.770617    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6beed4ed_3890_4845_952a_71bb8ed502bf.slice/crio-2ce83d51c5feda5abdc603f63d2760d5c4eed7fae2829c665754d600006329e7 WatchSource:0}: Error finding container 2ce83d51c5feda5abdc603f63d2760d5c4eed7fae2829c665754d600006329e7: Status 404 returned error can't find the container with id 2ce83d51c5feda5abdc603f63d2760d5c4eed7fae2829c665754d600006329e7
Nov 03 19:46:18.805530 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.805492    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:18.926419 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:18.926389    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.186361 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.186259    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.229180 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.229143    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-vcwk2"]
Nov 03 19:46:19.233541 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.233520    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.241106 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.241080    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 19:46:19.241236 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.241115    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-p8nbs\""
Nov 03 19:46:19.241236 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.241186    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 19:46:19.243919 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.243894    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vcwk2"]
Nov 03 19:46:19.314400 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.314368    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.359866 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.359829    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd62e6c4-72ba-44df-aab2-af25479da264-bound-sa-token\") pod \"cert-manager-597b95cb84-vcwk2\" (UID: \"dd62e6c4-72ba-44df-aab2-af25479da264\") " pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.360046 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.359914    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jph\" (UniqueName: \"kubernetes.io/projected/dd62e6c4-72ba-44df-aab2-af25479da264-kube-api-access-58jph\") pod \"cert-manager-597b95cb84-vcwk2\" (UID: \"dd62e6c4-72ba-44df-aab2-af25479da264\") " pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.435642 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.435602    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"]
Nov 03 19:46:19.439157 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.439094    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.452804 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.452778    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"]
Nov 03 19:46:19.460423 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.460396    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd62e6c4-72ba-44df-aab2-af25479da264-bound-sa-token\") pod \"cert-manager-597b95cb84-vcwk2\" (UID: \"dd62e6c4-72ba-44df-aab2-af25479da264\") " pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.460530 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.460426    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jph\" (UniqueName: \"kubernetes.io/projected/dd62e6c4-72ba-44df-aab2-af25479da264-kube-api-access-58jph\") pod \"cert-manager-597b95cb84-vcwk2\" (UID: \"dd62e6c4-72ba-44df-aab2-af25479da264\") " pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.470987 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.470966    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 19:46:19.471262 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.471235    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 19:46:19.471354 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.471260    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m9cc9\""
Nov 03 19:46:19.478608 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.478585    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jph\" (UniqueName: \"kubernetes.io/projected/dd62e6c4-72ba-44df-aab2-af25479da264-kube-api-access-58jph\") pod \"cert-manager-597b95cb84-vcwk2\" (UID: \"dd62e6c4-72ba-44df-aab2-af25479da264\") " pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.478907 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.478872    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.478969 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.478911    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd62e6c4-72ba-44df-aab2-af25479da264-bound-sa-token\") pod \"cert-manager-597b95cb84-vcwk2\" (UID: \"dd62e6c4-72ba-44df-aab2-af25479da264\") " pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.547602 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.547569    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vcwk2"
Nov 03 19:46:19.561231 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.561209    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f3f7bc-4ef1-43fb-83ff-36926bf67807-tmp\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.561343 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.561324    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284v6\" (UniqueName: \"kubernetes.io/projected/31f3f7bc-4ef1-43fb-83ff-36926bf67807-kube-api-access-284v6\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.561407 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.561391    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f3f7bc-4ef1-43fb-83ff-36926bf67807-tls-certs\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.611552 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.607756    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.662151 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.662111    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f3f7bc-4ef1-43fb-83ff-36926bf67807-tmp\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.662359 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.662165    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284v6\" (UniqueName: \"kubernetes.io/projected/31f3f7bc-4ef1-43fb-83ff-36926bf67807-kube-api-access-284v6\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.662359 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.662223    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f3f7bc-4ef1-43fb-83ff-36926bf67807-tls-certs\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.664389 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.664360    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f3f7bc-4ef1-43fb-83ff-36926bf67807-tmp\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.664562 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.664547    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31f3f7bc-4ef1-43fb-83ff-36926bf67807-tls-certs\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.678216 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.678176    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284v6\" (UniqueName: \"kubernetes.io/projected/31f3f7bc-4ef1-43fb-83ff-36926bf67807-kube-api-access-284v6\") pod \"kube-auth-proxy-6dd88dbbb7-j2xmd\" (UID: \"31f3f7bc-4ef1-43fb-83ff-36926bf67807\") " pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.741442 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.741402    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:19.753157 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.753125    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b" event={"ID":"6beed4ed-3890-4845-952a-71bb8ed502bf","Type":"ContainerStarted","Data":"2ce83d51c5feda5abdc603f63d2760d5c4eed7fae2829c665754d600006329e7"}
Nov 03 19:46:19.754323 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.754294    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"
Nov 03 19:46:19.833048 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.831823    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vcwk2"]
Nov 03 19:46:19.870752 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:19.870674    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:20.068340 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:20.067693    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd"]
Nov 03 19:46:20.074373 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:46:20.074339    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f3f7bc_4ef1_43fb_83ff_36926bf67807.slice/crio-a2f1bbb4f22cc6ad43fabf8fba2605dffaee1a514f684e048f7d80a04136a34e WatchSource:0}: Error finding container a2f1bbb4f22cc6ad43fabf8fba2605dffaee1a514f684e048f7d80a04136a34e: Status 404 returned error can't find the container with id a2f1bbb4f22cc6ad43fabf8fba2605dffaee1a514f684e048f7d80a04136a34e
Nov 03 19:46:20.760473 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:20.760429    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd" event={"ID":"31f3f7bc-4ef1-43fb-83ff-36926bf67807","Type":"ContainerStarted","Data":"a2f1bbb4f22cc6ad43fabf8fba2605dffaee1a514f684e048f7d80a04136a34e"}
Nov 03 19:46:20.762160 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:20.762133    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vcwk2" event={"ID":"dd62e6c4-72ba-44df-aab2-af25479da264","Type":"ContainerStarted","Data":"64c0b0a82d3fb5391a03b032cdfd16ad83a599b7d5ae5df8bcc72d938c9388a3"}
Nov 03 19:46:21.188151 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:21.188066    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:21.444887 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:21.444729    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:22.982119 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:22.982083    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:23.779825 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:23.779790    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd" event={"ID":"31f3f7bc-4ef1-43fb-83ff-36926bf67807","Type":"ContainerStarted","Data":"b5da88f6eb8e4531888dd5e3034032cd0fdbb52999a35c0a89b73bf74d515343"}
Nov 03 19:46:23.795765 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:23.795717    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dd88dbbb7-j2xmd" podStartSLOduration=1.385544046 podStartE2EDuration="4.795702817s" totalImagesPullingTime="3.410158771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:23.794917205 +0000 UTC m=+376.464118674" watchObservedRunningTime="2026-11-03 19:46:23.795702817 +0000 UTC m=+376.464904281"
Nov 03 19:46:24.707747 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:24.707715    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:24.785582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:24.785541    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vcwk2" event={"ID":"dd62e6c4-72ba-44df-aab2-af25479da264","Type":"ContainerStarted","Data":"a31184d65d4ecb65d1d60a005a1b3bafa04841979fd650d0eb4ae8b19e18691f"}
Nov 03 19:46:24.805690 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:24.805642    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-vcwk2" podStartSLOduration=1.94898639 podStartE2EDuration="5.805628214s" totalImagesPullingTime="3.856641824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:24.80490674 +0000 UTC m=+377.474108230" watchObservedRunningTime="2026-11-03 19:46:24.805628214 +0000 UTC m=+377.474829681"
Nov 03 19:46:24.836623 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:24.836589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:25.790522 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:25.790483    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b" event={"ID":"6beed4ed-3890-4845-952a-71bb8ed502bf","Type":"ContainerStarted","Data":"7da1f9d6607289f187db286895872170a2bfafbee2752483358fe04922de7528"}
Nov 03 19:46:25.806123 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:25.806071    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fbz8b" podStartSLOduration=1.703432899 podStartE2EDuration="7.806055299s" totalImagesPullingTime="6.1026224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:25.805780748 +0000 UTC m=+378.474982218" watchObservedRunningTime="2026-11-03 19:46:25.806055299 +0000 UTC m=+378.475256766"
Nov 03 19:46:29.713682 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:29.713647    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:29.903740 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:29.903706    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:30.240527 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:30.240488    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:30.359260 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:30.359181    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:30.611117 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:30.611045    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:30.724482 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:30.724451    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:31.613029 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:46:31.612993    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:46:32.200730 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:32.200695    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:32.310339 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:32.310305    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:36.207443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:36.207409    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:36.318129 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:36.318098    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.003908 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:40.003871    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.100510 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:40.100476    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.598265 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:40.598225    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:40.709649 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:40.709613    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:43.007263 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:43.007229    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:43.120732 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:43.120695    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:48.765816 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:48.765771    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:48.883832 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:48.883801    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:55.355846 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:55.355808    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:55.468668 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:55.468620    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:55.800828 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:55.800799    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:55.922129 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:46:55.922096    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:01.612362 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:47:01.612317    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:47:17.722096 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.722063    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.781467 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.781430    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rrtn2"]
Nov 03 19:47:17.807450 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.807419    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rrtn2"]
Nov 03 19:47:17.807604 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.807540    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:17.814448 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.814420    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 19:47:17.814448 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.814448    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hxvjr\""
Nov 03 19:47:17.823691 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.823666    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bbs7v"]
Nov 03 19:47:17.826996 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.826974    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:17.843004 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.842976    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bbs7v"]
Nov 03 19:47:17.854095 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.854066    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:17.887140 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.887103    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"]
Nov 03 19:47:17.889944 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.889923    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 19:47:17.889944 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.889940    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 19:47:17.890429 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.890414    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:17.905207 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.905168    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"]
Nov 03 19:47:17.914943 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.914917    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 19:47:17.915057 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.914979    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-86w7q\""
Nov 03 19:47:17.918848 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.918730    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6887d4ff-a9ab-423f-b73b-f681f1e8e474-tls-certs\") pod \"model-serving-api-75bd89fdfc-bbs7v\" (UID: \"6887d4ff-a9ab-423f-b73b-f681f1e8e474\") " pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:17.918848 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.918768    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bfd6f63-1e63-426c-9728-c3de74713bf1-cert\") pod \"kserve-controller-manager-776cc66d47-rrtn2\" (UID: \"8bfd6f63-1e63-426c-9728-c3de74713bf1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:17.918848 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.918803    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvtn\" (UniqueName: \"kubernetes.io/projected/8bfd6f63-1e63-426c-9728-c3de74713bf1-kube-api-access-vfvtn\") pod \"kserve-controller-manager-776cc66d47-rrtn2\" (UID: \"8bfd6f63-1e63-426c-9728-c3de74713bf1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:17.919055 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.919032    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggj9\" (UniqueName: \"kubernetes.io/projected/6887d4ff-a9ab-423f-b73b-f681f1e8e474-kube-api-access-zggj9\") pod \"model-serving-api-75bd89fdfc-bbs7v\" (UID: \"6887d4ff-a9ab-423f-b73b-f681f1e8e474\") " pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:17.985720 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:17.985687    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:18.020274 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020235    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/860c0078-098f-46a0-832d-507c6dc76dc8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.020438 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020282    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6887d4ff-a9ab-423f-b73b-f681f1e8e474-tls-certs\") pod \"model-serving-api-75bd89fdfc-bbs7v\" (UID: \"6887d4ff-a9ab-423f-b73b-f681f1e8e474\") " pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:18.020438 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020306    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bfd6f63-1e63-426c-9728-c3de74713bf1-cert\") pod \"kserve-controller-manager-776cc66d47-rrtn2\" (UID: \"8bfd6f63-1e63-426c-9728-c3de74713bf1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:18.020438 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020358    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvtn\" (UniqueName: \"kubernetes.io/projected/8bfd6f63-1e63-426c-9728-c3de74713bf1-kube-api-access-vfvtn\") pod \"kserve-controller-manager-776cc66d47-rrtn2\" (UID: \"8bfd6f63-1e63-426c-9728-c3de74713bf1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:18.020438 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020388    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvkd\" (UniqueName: \"kubernetes.io/projected/860c0078-098f-46a0-832d-507c6dc76dc8-kube-api-access-dqvkd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.020438 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020433    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.020642 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020463    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.020642 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020535    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860c0078-098f-46a0-832d-507c6dc76dc8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.020642 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020623    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggj9\" (UniqueName: \"kubernetes.io/projected/6887d4ff-a9ab-423f-b73b-f681f1e8e474-kube-api-access-zggj9\") pod \"model-serving-api-75bd89fdfc-bbs7v\" (UID: \"6887d4ff-a9ab-423f-b73b-f681f1e8e474\") " pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:18.020743 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020653    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/860c0078-098f-46a0-832d-507c6dc76dc8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.020743 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.020728    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.022752 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.022733    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bfd6f63-1e63-426c-9728-c3de74713bf1-cert\") pod \"kserve-controller-manager-776cc66d47-rrtn2\" (UID: \"8bfd6f63-1e63-426c-9728-c3de74713bf1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:18.022918 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.022898    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6887d4ff-a9ab-423f-b73b-f681f1e8e474-tls-certs\") pod \"model-serving-api-75bd89fdfc-bbs7v\" (UID: \"6887d4ff-a9ab-423f-b73b-f681f1e8e474\") " pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:18.027959 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.027938    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvtn\" (UniqueName: \"kubernetes.io/projected/8bfd6f63-1e63-426c-9728-c3de74713bf1-kube-api-access-vfvtn\") pod \"kserve-controller-manager-776cc66d47-rrtn2\" (UID: \"8bfd6f63-1e63-426c-9728-c3de74713bf1\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:18.028310 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.028280    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggj9\" (UniqueName: \"kubernetes.io/projected/6887d4ff-a9ab-423f-b73b-f681f1e8e474-kube-api-access-zggj9\") pod \"model-serving-api-75bd89fdfc-bbs7v\" (UID: \"6887d4ff-a9ab-423f-b73b-f681f1e8e474\") " pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:18.102937 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.102898    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:18.121902 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.121871    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:18.122052 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122010    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/860c0078-098f-46a0-832d-507c6dc76dc8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.122109 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122053    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvkd\" (UniqueName: \"kubernetes.io/projected/860c0078-098f-46a0-832d-507c6dc76dc8-kube-api-access-dqvkd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.122109 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122082    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.122109 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122103    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.122273 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122135    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860c0078-098f-46a0-832d-507c6dc76dc8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.122273 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122163    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/860c0078-098f-46a0-832d-507c6dc76dc8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.122273 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122229    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.122991 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.122955    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/860c0078-098f-46a0-832d-507c6dc76dc8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.124418 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.124390    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.124418 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.124404    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/860c0078-098f-46a0-832d-507c6dc76dc8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.124588 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.124464    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.124588 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.124529    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/860c0078-098f-46a0-832d-507c6dc76dc8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.133357 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.133337    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvkd\" (UniqueName: \"kubernetes.io/projected/860c0078-098f-46a0-832d-507c6dc76dc8-kube-api-access-dqvkd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.133601 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.133558    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/860c0078-098f-46a0-832d-507c6dc76dc8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-85svm\" (UID: \"860c0078-098f-46a0-832d-507c6dc76dc8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.140041 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.140010    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:18.201600 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.201378    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:18.397358 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.397185    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rrtn2"]
Nov 03 19:47:18.439065 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.438916    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bbs7v"]
Nov 03 19:47:18.441514 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:47:18.441488    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6887d4ff_a9ab_423f_b73b_f681f1e8e474.slice/crio-4c5bafe1fd67c0968f3f5b8a7527ba8082afd6bd3daccb34ec8b210640bb76a9 WatchSource:0}: Error finding container 4c5bafe1fd67c0968f3f5b8a7527ba8082afd6bd3daccb34ec8b210640bb76a9: Status 404 returned error can't find the container with id 4c5bafe1fd67c0968f3f5b8a7527ba8082afd6bd3daccb34ec8b210640bb76a9
Nov 03 19:47:18.473256 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.473220    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"]
Nov 03 19:47:18.476500 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:47:18.476435    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860c0078_098f_46a0_832d_507c6dc76dc8.slice/crio-5553f4e88e41407fbc304476629283ea62302f4df6de26f48345da623052e820 WatchSource:0}: Error finding container 5553f4e88e41407fbc304476629283ea62302f4df6de26f48345da623052e820: Status 404 returned error can't find the container with id 5553f4e88e41407fbc304476629283ea62302f4df6de26f48345da623052e820
Nov 03 19:47:18.478590 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.478554    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:18.478663 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.478619    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:47:18.857949 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.857842    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:18.985511 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:18.985458    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.001957 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.001908    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v" event={"ID":"6887d4ff-a9ab-423f-b73b-f681f1e8e474","Type":"ContainerStarted","Data":"4c5bafe1fd67c0968f3f5b8a7527ba8082afd6bd3daccb34ec8b210640bb76a9"}
Nov 03 19:47:19.004066 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.004037    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm" event={"ID":"860c0078-098f-46a0-832d-507c6dc76dc8","Type":"ContainerStarted","Data":"a178eb13bbaa7c4b6d9c3d56cf052c71c2fb0d301729296f034b9a911b4d732d"}
Nov 03 19:47:19.004191 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.004085    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm" event={"ID":"860c0078-098f-46a0-832d-507c6dc76dc8","Type":"ContainerStarted","Data":"5553f4e88e41407fbc304476629283ea62302f4df6de26f48345da623052e820"}
Nov 03 19:47:19.004259 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.004235    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:19.006076 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.006050    5282 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-85svm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:47:19.006226 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.006099    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm" podUID="860c0078-098f-46a0-832d-507c6dc76dc8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:47:19.010635 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.010608    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2" event={"ID":"8bfd6f63-1e63-426c-9728-c3de74713bf1","Type":"ContainerStarted","Data":"d5bfc7b14be2c6a875478778d659d00e59b84a3e7566e16d82ec4f35f6eda3b1"}
Nov 03 19:47:19.026555 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.026493    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm" podStartSLOduration=2.026473652 podStartE2EDuration="2.026473652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:19.024249798 +0000 UTC m=+431.693451268" watchObservedRunningTime="2026-11-03 19:47:19.026473652 +0000 UTC m=+431.695675120"
Nov 03 19:47:19.101792 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.101753    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:19.282208 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:19.282156    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:20.016061 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.016021    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-85svm"
Nov 03 19:47:20.068142 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.067723    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"]
Nov 03 19:47:20.069169 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.069133    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl" podUID="66b1a074-4d72-4490-b193-a4e3e8003047" containerName="discovery" containerID="cri-o://d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa" gracePeriod=30
Nov 03 19:47:20.360644 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.360615    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:47:20.446134 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.446099    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-kubeconfig\") pod \"66b1a074-4d72-4490-b193-a4e3e8003047\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") "
Nov 03 19:47:20.446134 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.446135    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-cacerts\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-cacerts\") pod \"66b1a074-4d72-4490-b193-a4e3e8003047\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") "
Nov 03 19:47:20.446375 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.446166    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-dns-cert\") pod \"66b1a074-4d72-4490-b193-a4e3e8003047\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") "
Nov 03 19:47:20.446375 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.446299    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-kube-api-access-x7lxn\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-kube-api-access-x7lxn\") pod \"66b1a074-4d72-4490-b193-a4e3e8003047\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") "
Nov 03 19:47:20.446466 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.446373    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66b1a074-4d72-4490-b193-a4e3e8003047-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1a074-4d72-4490-b193-a4e3e8003047-local-certs\") pod \"66b1a074-4d72-4490-b193-a4e3e8003047\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") "
Nov 03 19:47:20.446466 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.446401    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-istio-token\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-istio-token\") pod \"66b1a074-4d72-4490-b193-a4e3e8003047\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") "
Nov 03 19:47:20.446466 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.446430    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-ca-configmap\") pod \"66b1a074-4d72-4490-b193-a4e3e8003047\" (UID: \"66b1a074-4d72-4490-b193-a4e3e8003047\") "
Nov 03 19:47:20.447335 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.447267    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-ca-configmap" pod "66b1a074-4d72-4490-b193-a4e3e8003047" (UID: "66b1a074-4d72-4490-b193-a4e3e8003047"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:20.449539 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.449512    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-istio-token" pod "66b1a074-4d72-4490-b193-a4e3e8003047" (UID: "66b1a074-4d72-4490-b193-a4e3e8003047"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:20.449646 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.449534    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-cacerts" pod "66b1a074-4d72-4490-b193-a4e3e8003047" (UID: "66b1a074-4d72-4490-b193-a4e3e8003047"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:20.449646 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.449546    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1a074-4d72-4490-b193-a4e3e8003047-local-certs" pod "66b1a074-4d72-4490-b193-a4e3e8003047" (UID: "66b1a074-4d72-4490-b193-a4e3e8003047"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:20.449747 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.449640    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-dns-cert" pod "66b1a074-4d72-4490-b193-a4e3e8003047" (UID: "66b1a074-4d72-4490-b193-a4e3e8003047"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:20.449747 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.449689    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-kube-api-access-x7lxn" pod "66b1a074-4d72-4490-b193-a4e3e8003047" (UID: "66b1a074-4d72-4490-b193-a4e3e8003047"). InnerVolumeSpecName "kube-api-access-x7lxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:20.451447 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.451413    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-kubeconfig" pod "66b1a074-4d72-4490-b193-a4e3e8003047" (UID: "66b1a074-4d72-4490-b193-a4e3e8003047"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:20.547302 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.547262    5282 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-kubeconfig\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.547302 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.547294    5282 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-cacerts\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.547302 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.547308    5282 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-dns-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.547540 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.547324    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7lxn\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-kube-api-access-x7lxn\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.547540 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.547340    5282 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b1a074-4d72-4490-b193-a4e3e8003047-local-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.547540 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.547354    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b1a074-4d72-4490-b193-a4e3e8003047-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:20.547540 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:20.547368    5282 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66b1a074-4d72-4490-b193-a4e3e8003047-istio-csr-ca-configmap\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:21.024764 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.024730    5282 generic.go:356] "Generic (PLEG): container finished" podID="66b1a074-4d72-4490-b193-a4e3e8003047" containerID="d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa" exitCode=0
Nov 03 19:47:21.025229 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.024778    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"
Nov 03 19:47:21.025229 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.024801    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl" event={"ID":"66b1a074-4d72-4490-b193-a4e3e8003047","Type":"ContainerDied","Data":"d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa"}
Nov 03 19:47:21.025229 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.024869    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl" event={"ID":"66b1a074-4d72-4490-b193-a4e3e8003047","Type":"ContainerDied","Data":"dcb7e427ee90ee937228be9cfe56c00f3dce3d3591c8b1fd8877e5bb3887fc36"}
Nov 03 19:47:21.025229 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.024900    5282 scope.go:122] "RemoveContainer" containerID="d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa"
Nov 03 19:47:21.027084 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.026646    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v" event={"ID":"6887d4ff-a9ab-423f-b73b-f681f1e8e474","Type":"ContainerStarted","Data":"4d710c418c57d1bf6c69291056b27c41f88e01e77579040e5ec83d39250e36a0"}
Nov 03 19:47:21.027084 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.026713    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:21.028408 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.028383    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2" event={"ID":"8bfd6f63-1e63-426c-9728-c3de74713bf1","Type":"ContainerStarted","Data":"2826762fbd3d5a7d3e9ad799e8885102ef05da056865d2f71e03f19fc9362e64"}
Nov 03 19:47:21.028523 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.028457    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:47:21.035503 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.035485    5282 scope.go:122] "RemoveContainer" containerID="d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa"
Nov 03 19:47:21.035775 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:47:21.035750    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa\": container with ID starting with d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa not found: ID does not exist" containerID="d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa"
Nov 03 19:47:21.035821 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.035784    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa"} err="failed to get container status \"d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa\": rpc error: code = NotFound desc = could not find container \"d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa\": container with ID starting with d8da567d291766b0b7170c537c63ca25f93191c09e363db198bb12a2c21c6ffa not found: ID does not exist"
Nov 03 19:47:21.046828 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.046747    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2" podStartSLOduration=1.742622936 podStartE2EDuration="4.04673417s" totalImagesPullingTime="2.304111234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:21.046442549 +0000 UTC m=+433.715644020" watchObservedRunningTime="2026-11-03 19:47:21.04673417 +0000 UTC m=+433.715935638"
Nov 03 19:47:21.063324 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.063273    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v" podStartSLOduration=1.790420345 podStartE2EDuration="4.063259265s" totalImagesPullingTime="2.27283892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:21.062597739 +0000 UTC m=+433.731799213" watchObservedRunningTime="2026-11-03 19:47:21.063259265 +0000 UTC m=+433.732460735"
Nov 03 19:47:21.075554 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.075515    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"]
Nov 03 19:47:21.081771 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.081744    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-b8qdl"]
Nov 03 19:47:21.452936 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:21.452858    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b1a074-4d72-4490-b193-a4e3e8003047" path="/var/lib/kubelet/pods/66b1a074-4d72-4490-b193-a4e3e8003047/volumes"
Nov 03 19:47:31.612687 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:47:31.612648    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:47:32.037519 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:32.037489    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-bbs7v"
Nov 03 19:47:52.039096 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:47:52.039018    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rrtn2"
Nov 03 19:48:01.613460 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:48:01.613418    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:48:10.921462 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:48:10.921429    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:11.036627 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:48:11.036587    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:14.206086 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:48:14.206033    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:14.327547 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:48:14.327513    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:19.312504 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:48:19.312434    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:31.612744 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:48:31.612701    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:49:01.613092 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:49:01.613046    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:49:19.307552 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:49:19.307453    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:19.514204 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:49:19.514163    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:19.636793 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:49:19.636718    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:31.612469 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:49:31.612429    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:50:01.612997 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:50:01.612960    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:50:15.549347 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:50:15.549313    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:19.395324 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:50:19.395288    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:31.613045 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:50:31.613006    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:51:01.612687 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:51:01.612601    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:51:19.452335 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:51:19.452300    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:20.705451 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:51:20.705417    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:20.809800 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:51:20.809768    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:31.612781 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:51:31.612738    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:52:01.612339 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:52:01.612302    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:52:19.493116 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:52:19.493032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:31.612816 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:52:31.612778    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:52:32.099876 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:52:32.099833    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:35.532561 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:52:35.532526    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:01.611958 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:53:01.611917    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:53:19.544234 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:53:19.544184    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:31.612244 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:53:31.612188    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:54:01.612871 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:54:01.612791    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:54:19.584688 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:54:19.584650    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:31.613378 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:54:31.613341    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:55:01.612961 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:55:01.612922    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:55:19.583244 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:55:19.583130    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:55:31.612779 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:55:31.612743    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:55:36.092219 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:55:36.092155    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:01.612787 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:56:01.612752    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:56:19.678767 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:19.678732    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:31.612116 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:56:31.612080    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:56:57.021040 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.020990    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.139959 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.139922    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.379225 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.379127    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:56:57.379448 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.379436    5282 cpu_manager.go:414] "containerMap: removing container" podUID="66b1a074-4d72-4490-b193-a4e3e8003047" containerName="discovery"
Nov 03 19:56:57.379501 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.379450    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b1a074-4d72-4490-b193-a4e3e8003047" containerName="discovery"
Nov 03 19:56:57.379540 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.379506    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b1a074-4d72-4490-b193-a4e3e8003047" containerName="discovery"
Nov 03 19:56:57.382394 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.382375    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.385791 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.385771    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 19:56:57.391176 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.391157    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:56:57.420504 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.420475    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:56:57.505456 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.505421    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4159e7e-3404-4698-9fba-aea080862989-config-file\") pod \"limitador-limitador-7c9b87d574-lm5fq\" (UID: \"c4159e7e-3404-4698-9fba-aea080862989\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.505613 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.505466    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb744\" (UniqueName: \"kubernetes.io/projected/c4159e7e-3404-4698-9fba-aea080862989-kube-api-access-lb744\") pod \"limitador-limitador-7c9b87d574-lm5fq\" (UID: \"c4159e7e-3404-4698-9fba-aea080862989\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.606166 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.606132    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4159e7e-3404-4698-9fba-aea080862989-config-file\") pod \"limitador-limitador-7c9b87d574-lm5fq\" (UID: \"c4159e7e-3404-4698-9fba-aea080862989\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.606355 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.606178    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb744\" (UniqueName: \"kubernetes.io/projected/c4159e7e-3404-4698-9fba-aea080862989-kube-api-access-lb744\") pod \"limitador-limitador-7c9b87d574-lm5fq\" (UID: \"c4159e7e-3404-4698-9fba-aea080862989\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.606785 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.606765    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c4159e7e-3404-4698-9fba-aea080862989-config-file\") pod \"limitador-limitador-7c9b87d574-lm5fq\" (UID: \"c4159e7e-3404-4698-9fba-aea080862989\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.613851 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.613829    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb744\" (UniqueName: \"kubernetes.io/projected/c4159e7e-3404-4698-9fba-aea080862989-kube-api-access-lb744\") pod \"limitador-limitador-7c9b87d574-lm5fq\" (UID: \"c4159e7e-3404-4698-9fba-aea080862989\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.695927 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.695843    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:56:57.852663 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.852625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.963582 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.963506    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:56:57.966654 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:56:57.966622    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4159e7e_3404_4698_9fba_aea080862989.slice/crio-481698e206cc1dfa33e057835b72f0798d6179e8b6f6c29ebbe5d6ff98248249 WatchSource:0}: Error finding container 481698e206cc1dfa33e057835b72f0798d6179e8b6f6c29ebbe5d6ff98248249: Status 404 returned error can't find the container with id 481698e206cc1dfa33e057835b72f0798d6179e8b6f6c29ebbe5d6ff98248249
Nov 03 19:56:57.966770 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.966639    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:57.968689 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:57.968675    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:56:58.293589 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:56:58.293550    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq" event={"ID":"c4159e7e-3404-4698-9fba-aea080862989","Type":"ContainerStarted","Data":"481698e206cc1dfa33e057835b72f0798d6179e8b6f6c29ebbe5d6ff98248249"}
Nov 03 19:57:01.307694 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:01.307659    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq" event={"ID":"c4159e7e-3404-4698-9fba-aea080862989","Type":"ContainerStarted","Data":"67133115fe2a3b5e0d433123ddcf4545deb4972ba1471ed2aa26bfdd3acbe5fc"}
Nov 03 19:57:01.308098 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:01.307788    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:57:01.322554 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:01.322511    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq" podStartSLOduration=1.574575073 podStartE2EDuration="4.3224988s" totalImagesPullingTime="2.747923727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:01.321979728 +0000 UTC m=+1013.991181197" watchObservedRunningTime="2026-11-03 19:57:01.3224988 +0000 UTC m=+1013.991700252"
Nov 03 19:57:01.612910 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:57:01.612825    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:57:12.312416 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:12.312385    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"
Nov 03 19:57:18.121770 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:18.121738    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:18.282502 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:18.282462    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:18.888030 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:18.887992    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:19.730215 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:19.730174    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:31.612723 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:57:31.612682    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:57:38.266262 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:38.266234    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:38.395586 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:38.395548    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:38.509298 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:38.509264    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:38.646027 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:38.645950    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:42.410293 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:42.410255    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:42.531848 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:42.531814    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:43.491521 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:43.491487    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:43.852771 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:43.852693    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:57:44.041172 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:57:44.041136    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:01.613108 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:58:01.613068    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:58:14.237702 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:14.237665    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:14.376368 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:14.376329    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.493325 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:15.493294    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.627436 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:15.627398    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.737347 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:15.737317    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:15.856423 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:15.856346    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:16.348956 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:16.348916    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:58:19.790381 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:19.790296    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:25.784462 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:25.784423    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:25.890142 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:25.890110    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:58:26.274544 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.274509    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"]
Nov 03 19:58:26.277700 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.277675    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.288152 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.288133    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:58:26.288305 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.288249    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:58:26.289464 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.289439    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"]
Nov 03 19:58:26.303568 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.303538    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 19:58:26.303568 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.303543    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:58:26.303750 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.303587    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7bzpm\""
Nov 03 19:58:26.416679 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.416647    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-metrics-certs\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.416679 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.416691    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-maas-api-tls\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.416903 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.416719    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5fls\" (UniqueName: \"kubernetes.io/projected/0ce07bb7-3be0-4e7c-83ce-e12206831846-kube-api-access-q5fls\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.416903 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.416775    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce07bb7-3be0-4e7c-83ce-e12206831846-openshift-service-ca\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.517436 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.517406    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-metrics-certs\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.517610 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.517449    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-maas-api-tls\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.517610 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.517476    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5fls\" (UniqueName: \"kubernetes.io/projected/0ce07bb7-3be0-4e7c-83ce-e12206831846-kube-api-access-q5fls\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.517610 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.517498    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce07bb7-3be0-4e7c-83ce-e12206831846-openshift-service-ca\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.518293 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.518266    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce07bb7-3be0-4e7c-83ce-e12206831846-openshift-service-ca\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.519847 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.519826    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-metrics-certs\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.519959 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.519941    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-maas-api-tls\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.526019 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.525956    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5fls\" (UniqueName: \"kubernetes.io/projected/0ce07bb7-3be0-4e7c-83ce-e12206831846-kube-api-access-q5fls\") pod \"maas-api-5cb65579d5-gj67g\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") " pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.590351 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.590318    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:26.862373 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:26.862227    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"]
Nov 03 19:58:26.865031 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:58:26.865000    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce07bb7_3be0_4e7c_83ce_e12206831846.slice/crio-c61154b1748c22a9c08ebfe27322a9566a7f473084b502c26a7cb0899043e68d WatchSource:0}: Error finding container c61154b1748c22a9c08ebfe27322a9566a7f473084b502c26a7cb0899043e68d: Status 404 returned error can't find the container with id c61154b1748c22a9c08ebfe27322a9566a7f473084b502c26a7cb0899043e68d
Nov 03 19:58:27.640447 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:27.640412    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g" event={"ID":"0ce07bb7-3be0-4e7c-83ce-e12206831846","Type":"ContainerStarted","Data":"c61154b1748c22a9c08ebfe27322a9566a7f473084b502c26a7cb0899043e68d"}
Nov 03 19:58:29.649923 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:29.649885    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g" event={"ID":"0ce07bb7-3be0-4e7c-83ce-e12206831846","Type":"ContainerStarted","Data":"651f253221af677ca3e48f5397600e4ced1b467e1247d07acb44ac27c5304d32"}
Nov 03 19:58:29.650441 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:29.650030    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:29.667700 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:29.667652    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g" podStartSLOduration=1.095050605 podStartE2EDuration="3.6676402s" totalImagesPullingTime="2.572589595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:29.666869224 +0000 UTC m=+1102.336070709" watchObservedRunningTime="2026-11-03 19:58:29.6676402 +0000 UTC m=+1102.336841670"
Nov 03 19:58:31.612275 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:58:31.612233    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:58:35.659476 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:35.659446    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 19:58:45.030060 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.030020    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"]
Nov 03 19:58:45.033626 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.033602    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.039684 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.039653    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dkm76\""
Nov 03 19:58:45.050734 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.050701    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"]
Nov 03 19:58:45.071829 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.071788    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072009 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.071841    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072009 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.071917    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072106 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.072015    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072106 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.072060    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072170 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.072129    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072170 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.072164    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072262 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.072185    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwfd\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-kube-api-access-swwfd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.072312 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.072298    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.173844 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.173363    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.174100 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.174080    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.174589 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.174571    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.175068 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.175051    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.175518 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.175500    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.175994 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.175976    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.176872 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.176854    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.177290 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.177273    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwfd\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-kube-api-access-swwfd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.177722 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.177701    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.177925 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.175419    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.178074 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.177217    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.178074 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.176783    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.178074 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.173821    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.178074 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.174515    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.179714 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.179692    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.180311 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.180290    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.187980 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.187949    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.188238 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.188187    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwfd\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-kube-api-access-swwfd\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kpkjt\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.346448 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.346343    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:45.620899 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.620808    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"]
Nov 03 19:58:45.623900 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:58:45.623869    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-29904f8bcccf3e0d37ff906c45c7da804929af7e971a17a84d626e08786550e6 WatchSource:0}: Error finding container 29904f8bcccf3e0d37ff906c45c7da804929af7e971a17a84d626e08786550e6: Status 404 returned error can't find the container with id 29904f8bcccf3e0d37ff906c45c7da804929af7e971a17a84d626e08786550e6
Nov 03 19:58:45.713540 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:45.713505    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" event={"ID":"ab4e8061-b105-45fc-8a9e-a35605b4bcb7","Type":"ContainerStarted","Data":"29904f8bcccf3e0d37ff906c45c7da804929af7e971a17a84d626e08786550e6"}
Nov 03 19:58:49.361678 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:49.361646    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:49.361929 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:49.361718    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:49.361929 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:49.361755    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:58:49.732079 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:49.732047    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" event={"ID":"ab4e8061-b105-45fc-8a9e-a35605b4bcb7","Type":"ContainerStarted","Data":"796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1"}
Nov 03 19:58:49.764043 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:49.763281    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podStartSLOduration=1.028339926 podStartE2EDuration="4.763262346s" totalImagesPullingTime="3.73492242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:49.762629153 +0000 UTC m=+1122.431830652" watchObservedRunningTime="2026-11-03 19:58:49.763262346 +0000 UTC m=+1122.432463819"
Nov 03 19:58:50.346653 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:50.346612    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:50.348078 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:50.348054    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 19:58:50.348184 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:50.348099    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 19:58:51.347115 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:51.347070    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 19:58:51.347538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:51.347131    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 19:58:52.347078 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:52.347032    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 19:58:52.347300 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:52.347090    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 19:58:53.347628 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:53.347588    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 19:58:53.348036 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:53.347644    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 19:58:54.347367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:54.347328    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Nov 03 19:58:54.347550 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:54.347392    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Nov 03 19:58:55.351341 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:55.351300    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:55.351823 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:55.351597    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:58:55.352312 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:58:55.352295    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 19:59:00.047868 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.047832    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"]
Nov 03 19:59:00.051068 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.051052    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.063310 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.063283    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"]
Nov 03 19:59:00.082093 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.082064    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"]
Nov 03 19:59:00.085443 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.085417    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.099456 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.099331    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"]
Nov 03 19:59:00.113455 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113419    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113592 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113498    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113592 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113541    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29w8n\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-kube-api-access-29w8n\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113674 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113623    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35226e43-d541-43f8-a888-edb3202ad0f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113674 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113656    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35226e43-d541-43f8-a888-edb3202ad0f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113752 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113676    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113752 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113735    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113822 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113772    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.113822 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.113792    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.214877 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.214845    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.214877 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.214882    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.215123 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.214903    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.215123 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.214921    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.215123 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.214976    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29w8n\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-kube-api-access-29w8n\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215123 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215114    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.215367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215160    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdkcg\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-kube-api-access-jdkcg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.215367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215186    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35226e43-d541-43f8-a888-edb3202ad0f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215234    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35226e43-d541-43f8-a888-edb3202ad0f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215285    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215316    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215321    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215367 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215333    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.215716 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215495    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215716 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215514    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215716 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215535    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.215716 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215555    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.215716 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215654    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.216118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215765    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.216118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215795    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.216118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215846    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.216118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215874    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.216118 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.215925    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35226e43-d541-43f8-a888-edb3202ad0f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.217661 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.217638    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.217751 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.217684    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35226e43-d541-43f8-a888-edb3202ad0f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.222755 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.222733    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29w8n\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-kube-api-access-29w8n\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.222985 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.222963    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2n8gb\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.317309 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317224    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdkcg\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-kube-api-access-jdkcg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317309 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317288    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317315    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317331    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317346    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317379    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317403    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317427    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317538 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317457    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317889 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317802    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317889 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317850    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.317995 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317884    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.318052 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.317995    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.318298 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.318277    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.319882 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.319865    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.319930 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.319906    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.324803 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.324770    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.325022 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.325001    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdkcg\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-kube-api-access-jdkcg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-7krc7\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.363872 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.363847    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:00.400703 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.400459    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:00.638254 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.638192    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"]
Nov 03 19:59:00.640513 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:59:00.640482    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35226e43_d541_43f8_a888_edb3202ad0f6.slice/crio-07d7f8c4d11cff94010d44a86211ffb708e184b274a7cd9c01fac838c2c30abf WatchSource:0}: Error finding container 07d7f8c4d11cff94010d44a86211ffb708e184b274a7cd9c01fac838c2c30abf: Status 404 returned error can't find the container with id 07d7f8c4d11cff94010d44a86211ffb708e184b274a7cd9c01fac838c2c30abf
Nov 03 19:59:00.643526 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.643493    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:00.643623 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.643557    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:00.643623 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.643604    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:00.679995 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.679962    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"]
Nov 03 19:59:00.684121 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:59:00.684097    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cf0dfd_822a_451b_8c1d_e2c3ce62f7f4.slice/crio-c72bd7329508f657651888aa76508a2a2f9b33736c95498ae401145052bc2e92 WatchSource:0}: Error finding container c72bd7329508f657651888aa76508a2a2f9b33736c95498ae401145052bc2e92: Status 404 returned error can't find the container with id c72bd7329508f657651888aa76508a2a2f9b33736c95498ae401145052bc2e92
Nov 03 19:59:00.687165 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.687128    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:00.687275 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.687230    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:00.687338 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.687280    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 19:59:00.788225 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.784761    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" event={"ID":"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4","Type":"ContainerStarted","Data":"caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a"}
Nov 03 19:59:00.788225 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.784822    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" event={"ID":"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4","Type":"ContainerStarted","Data":"c72bd7329508f657651888aa76508a2a2f9b33736c95498ae401145052bc2e92"}
Nov 03 19:59:00.788225 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.787434    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" event={"ID":"35226e43-d541-43f8-a888-edb3202ad0f6","Type":"ContainerStarted","Data":"93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a"}
Nov 03 19:59:00.788225 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.787482    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" event={"ID":"35226e43-d541-43f8-a888-edb3202ad0f6","Type":"ContainerStarted","Data":"07d7f8c4d11cff94010d44a86211ffb708e184b274a7cd9c01fac838c2c30abf"}
Nov 03 19:59:00.807493 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.807359    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podStartSLOduration=0.807274626 podStartE2EDuration="807.274626ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:00.804790493 +0000 UTC m=+1133.473991956" watchObservedRunningTime="2026-11-03 19:59:00.807274626 +0000 UTC m=+1133.476476092"
Nov 03 19:59:00.825442 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:00.824847    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podStartSLOduration=0.824827307 podStartE2EDuration="824.827307ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:00.823906793 +0000 UTC m=+1133.493108265" watchObservedRunningTime="2026-11-03 19:59:00.824827307 +0000 UTC m=+1133.494028780"
Nov 03 19:59:01.365173 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:01.365131    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:01.366524 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:01.366495    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 03 19:59:01.366661 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:01.366554    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 03 19:59:01.404648 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:01.404613    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:01.406633 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:01.406599    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 19:59:01.406883 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:01.406851    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 19:59:01.644112 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:59:01.644020    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:59:02.365280 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:02.365245    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 03 19:59:02.365697 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:02.365301    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 03 19:59:02.402223 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:02.401806    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 19:59:02.402223 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:02.401897    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 19:59:02.524177 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:02.519243    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:02.693945 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:02.693776    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:03.364976 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:03.364942    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 03 19:59:03.365258 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:03.365233    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 03 19:59:03.405734 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:03.405696    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 19:59:03.406364 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:03.406331    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 19:59:04.364227 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:04.364171    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 03 19:59:04.364401 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:04.364240    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 03 19:59:04.402635 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:04.402599    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 19:59:04.402919 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:04.402888    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 19:59:05.364906 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:05.364872    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 03 19:59:05.365316 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:05.364929    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 03 19:59:05.403608 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:05.403570    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 19:59:05.403891 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:05.403864    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 19:59:06.365212 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:06.365167    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Nov 03 19:59:06.365734 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:06.365252    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Nov 03 19:59:06.404224 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:06.403648    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Nov 03 19:59:06.404224 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:06.403713    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Nov 03 19:59:07.368979 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:07.368948    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:07.369384 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:07.369234    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:07.370018 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:07.370000    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 19:59:07.405486 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:07.405448    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:07.406030 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:07.405979    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:07.407002 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:07.406981    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 19:59:19.838727 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:19.838694    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.491567 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:23.491532    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:23.903584 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:23.903509    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:59:25.561091 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.561060    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"]
Nov 03 19:59:25.564459 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.564439    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.567787 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.567764    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 19:59:25.568757 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.568733    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2z695\""
Nov 03 19:59:25.568757 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.568737    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 19:59:25.568905 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.568787    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 19:59:25.575139 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.575119    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"]
Nov 03 19:59:25.632249 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.632223    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:25.652752 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.652730    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.652865 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.652763    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.652865 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.652789    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/04e563bf-735c-4d94-9de1-d04994490f4b-kube-api-access-kn99b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.652934 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.652864    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e563bf-735c-4d94-9de1-d04994490f4b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.652978 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.652965    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.653014 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.653002    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.753581 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.753544    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/04e563bf-735c-4d94-9de1-d04994490f4b-kube-api-access-kn99b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.753581 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.753581    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e563bf-735c-4d94-9de1-d04994490f4b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.753825 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.753623    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.753825 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.753651    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.753825 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.753677    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.753825 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.753710    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.754093 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.754068    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.754149 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.754135    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.754186 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.754164    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.756038 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.756010    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04e563bf-735c-4d94-9de1-d04994490f4b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.756303 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.756286    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e563bf-735c-4d94-9de1-d04994490f4b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.766760 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.766741    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn99b\" (UniqueName: \"kubernetes.io/projected/04e563bf-735c-4d94-9de1-d04994490f4b-kube-api-access-kn99b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl\" (UID: \"04e563bf-735c-4d94-9de1-d04994490f4b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:25.877858 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:25.877774    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:26.131542 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:26.131471    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:59:26.149321 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:26.149291    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"]
Nov 03 19:59:26.152346 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:59:26.152316    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e563bf_735c_4d94_9de1_d04994490f4b.slice/crio-745be726490bb0d22e6a29cf3288988af729b0118bf7e40c11bbf5e25af3b54a WatchSource:0}: Error finding container 745be726490bb0d22e6a29cf3288988af729b0118bf7e40c11bbf5e25af3b54a: Status 404 returned error can't find the container with id 745be726490bb0d22e6a29cf3288988af729b0118bf7e40c11bbf5e25af3b54a
Nov 03 19:59:26.955613 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:26.955567    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl" event={"ID":"04e563bf-735c-4d94-9de1-d04994490f4b","Type":"ContainerStarted","Data":"745be726490bb0d22e6a29cf3288988af729b0118bf7e40c11bbf5e25af3b54a"}
Nov 03 19:59:27.961723 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:27.961684    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl" event={"ID":"04e563bf-735c-4d94-9de1-d04994490f4b","Type":"ContainerStarted","Data":"5ffc8eb318197ed3787d2b42d50b47048e6d17ea166da7bc564e1097b7435d83"}
Nov 03 19:59:27.978010 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:27.977962    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl" podStartSLOduration=1.370104436 podStartE2EDuration="2.977949835s" totalImagesPullingTime="1.607845399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:27.977318951 +0000 UTC m=+1160.646520421" watchObservedRunningTime="2026-11-03 19:59:27.977949835 +0000 UTC m=+1160.647151315"
Nov 03 19:59:28.347312 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:28.347276    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:28.606017 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:28.601135    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:59:28.878190 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:28.878100    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:31.003223 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:30.993265    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:59:31.615036 ip-10-0-142-244 kubenswrapper[5282]: E1103 19:59:31.614991    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 19:59:31.628211 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.628152    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"]
Nov 03 19:59:31.631776 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.631756    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.636014 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.635989    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 19:59:31.643095 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.643072    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"]
Nov 03 19:59:31.682644 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.682595    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:31.738982 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.738949    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.738982 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.738981    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.739222 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.739009    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.739222 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.739080    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp696\" (UniqueName: \"kubernetes.io/projected/95213a91-c3dd-4195-bcea-9e329e10679b-kube-api-access-pp696\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.739222 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.739128    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.739222 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.739151    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95213a91-c3dd-4195-bcea-9e329e10679b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.840552 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.840510    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.840552 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.840553    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.840818 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.840589    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.840818 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.840611    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp696\" (UniqueName: \"kubernetes.io/projected/95213a91-c3dd-4195-bcea-9e329e10679b-kube-api-access-pp696\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.840818 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.840645    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.840818 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.840798    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95213a91-c3dd-4195-bcea-9e329e10679b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.841082 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.841063    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.841140 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.841108    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.841140 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.841122    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.842805 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.842780    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95213a91-c3dd-4195-bcea-9e329e10679b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.843109 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.843091    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95213a91-c3dd-4195-bcea-9e329e10679b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.848466 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.848442    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp696\" (UniqueName: \"kubernetes.io/projected/95213a91-c3dd-4195-bcea-9e329e10679b-kube-api-access-pp696\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b\" (UID: \"95213a91-c3dd-4195-bcea-9e329e10679b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:31.945745 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:31.945644    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:32.220985 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:32.220946    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"]
Nov 03 19:59:32.223785 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:59:32.223750    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95213a91_c3dd_4195_bcea_9e329e10679b.slice/crio-5a6b75637590d62daf96911e5c238098d18fc5a314ed11e82c647a5646f66e07 WatchSource:0}: Error finding container 5a6b75637590d62daf96911e5c238098d18fc5a314ed11e82c647a5646f66e07: Status 404 returned error can't find the container with id 5a6b75637590d62daf96911e5c238098d18fc5a314ed11e82c647a5646f66e07
Nov 03 19:59:32.752170 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:32.752132    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:33.006918 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:33.006795    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b" event={"ID":"95213a91-c3dd-4195-bcea-9e329e10679b","Type":"ContainerStarted","Data":"f063ddd3aa1c5814b52d708917405f2a07378d19044e15c6eb68e5f7c1bd32e8"}
Nov 03 19:59:33.006918 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:33.006878    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b" event={"ID":"95213a91-c3dd-4195-bcea-9e329e10679b","Type":"ContainerStarted","Data":"5a6b75637590d62daf96911e5c238098d18fc5a314ed11e82c647a5646f66e07"}
Nov 03 19:59:33.044220 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:33.040924    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b" podStartSLOduration=1.719325187 podStartE2EDuration="2.040904289s" totalImagesPullingTime="321.579102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:33.0405999 +0000 UTC m=+1165.709801371" watchObservedRunningTime="2026-11-03 19:59:33.040904289 +0000 UTC m=+1165.710105772"
Nov 03 19:59:33.762531 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:33.762493    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:59:33.819012 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:33.818972    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:33.946219 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:33.946172    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:35.038223 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.030464    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"]
Nov 03 19:59:35.048979 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.048954    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"]
Nov 03 19:59:35.049261 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.049213    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.057517 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.056733    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 19:59:35.147137 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.147111    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:35.188792 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.188742    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.189053 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.189030    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8jg\" (UniqueName: \"kubernetes.io/projected/3bf6a00b-f4b7-4909-8155-e849fecd35ad-kube-api-access-sh8jg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.189233 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.189191    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.189434 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.189401    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf6a00b-f4b7-4909-8155-e849fecd35ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.189709 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.189689    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.189902 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.189869    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.290803 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.290719    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.291067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.291047    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.291232 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.291214    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.291355 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.291338    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8jg\" (UniqueName: \"kubernetes.io/projected/3bf6a00b-f4b7-4909-8155-e849fecd35ad-kube-api-access-sh8jg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.291469 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.291455    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.291604 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.291585    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf6a00b-f4b7-4909-8155-e849fecd35ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.292610 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.292588    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.292785 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.292765    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.293171 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.292955    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.314722 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.314694    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf6a00b-f4b7-4909-8155-e849fecd35ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.315519 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.315495    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8jg\" (UniqueName: \"kubernetes.io/projected/3bf6a00b-f4b7-4909-8155-e849fecd35ad-kube-api-access-sh8jg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.331216 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.330139    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf6a00b-f4b7-4909-8155-e849fecd35ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-44wlb\" (UID: \"3bf6a00b-f4b7-4909-8155-e849fecd35ad\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.372746 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.372711    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:35.784631 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.784459    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"]
Nov 03 19:59:35.787389 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:59:35.787364    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf6a00b_f4b7_4909_8155_e849fecd35ad.slice/crio-12503f8baa56856cb10fab0987018e2b553c20d5a81cdc18cfbb6b111fdcdbc3 WatchSource:0}: Error finding container 12503f8baa56856cb10fab0987018e2b553c20d5a81cdc18cfbb6b111fdcdbc3: Status 404 returned error can't find the container with id 12503f8baa56856cb10fab0987018e2b553c20d5a81cdc18cfbb6b111fdcdbc3
Nov 03 19:59:35.878807 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.878730    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:35.892266 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:35.892231    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:36.053150 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:36.053110    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb" event={"ID":"3bf6a00b-f4b7-4909-8155-e849fecd35ad","Type":"ContainerStarted","Data":"12503f8baa56856cb10fab0987018e2b553c20d5a81cdc18cfbb6b111fdcdbc3"}
Nov 03 19:59:36.064607 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:36.064582    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl"
Nov 03 19:59:36.836723 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:36.836685    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:37.060963 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:37.060920    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb" event={"ID":"3bf6a00b-f4b7-4909-8155-e849fecd35ad","Type":"ContainerStarted","Data":"4f57ed0fed847bd4def504591933005f545fe1931877ef96d061dd6219b550a9"}
Nov 03 19:59:37.109856 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:37.109715    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb" podStartSLOduration=1.806438955 podStartE2EDuration="2.109697144s" totalImagesPullingTime="303.258189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:37.109555838 +0000 UTC m=+1169.778757313" watchObservedRunningTime="2026-11-03 19:59:37.109697144 +0000 UTC m=+1169.778898615"
Nov 03 19:59:37.373763 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:37.373686    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:39.756546 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:39.756512    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 19:59:41.946096 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:41.946047    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:41.961110 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:41.961085    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:42.151689 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:42.151653    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b"
Nov 03 19:59:45.373138 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:45.373098    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:45.385566 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:45.385542    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:46.138924 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:46.138898    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb"
Nov 03 19:59:52.113847 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.113809    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-gtfjk"]
Nov 03 19:59:52.119067 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.119046    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.126574 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.126549    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 19:59:52.126690 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.126569    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-wcnrg\""
Nov 03 19:59:52.126774 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.126563    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 19:59:52.127053 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.127025    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-gtfjk"]
Nov 03 19:59:52.149148 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.149121    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:52.214470 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.214437    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764546a5-e946-4e8d-baf1-4126384304f0-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-gtfjk\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") " pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.214614 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.214478    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgrx9\" (UniqueName: \"kubernetes.io/projected/764546a5-e946-4e8d-baf1-4126384304f0-kube-api-access-jgrx9\") pod \"ai-gateway-controller-86c7698f44-gtfjk\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") " pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.271086 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.271053    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:52.315035 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.315005    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764546a5-e946-4e8d-baf1-4126384304f0-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-gtfjk\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") " pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.315216 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.315046    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgrx9\" (UniqueName: \"kubernetes.io/projected/764546a5-e946-4e8d-baf1-4126384304f0-kube-api-access-jgrx9\") pod \"ai-gateway-controller-86c7698f44-gtfjk\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") " pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.317444 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.317421    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764546a5-e946-4e8d-baf1-4126384304f0-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-gtfjk\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") " pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.322549 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.322520    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgrx9\" (UniqueName: \"kubernetes.io/projected/764546a5-e946-4e8d-baf1-4126384304f0-kube-api-access-jgrx9\") pod \"ai-gateway-controller-86c7698f44-gtfjk\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") " pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.441775 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.441697    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:52.708845 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:52.708816    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-gtfjk"]
Nov 03 19:59:52.711343 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:59:52.711316    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764546a5_e946_4e8d_baf1_4126384304f0.slice/crio-a06beea3c4ccf4bbee4c9bc38608b7f56590d1f5da1684eda167b095bc84e41c WatchSource:0}: Error finding container a06beea3c4ccf4bbee4c9bc38608b7f56590d1f5da1684eda167b095bc84e41c: Status 404 returned error can't find the container with id a06beea3c4ccf4bbee4c9bc38608b7f56590d1f5da1684eda167b095bc84e41c
Nov 03 19:59:53.161974 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:53.161931    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk" event={"ID":"764546a5-e946-4e8d-baf1-4126384304f0","Type":"ContainerStarted","Data":"a06beea3c4ccf4bbee4c9bc38608b7f56590d1f5da1684eda167b095bc84e41c"}
Nov 03 19:59:57.185362 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.185275    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk" event={"ID":"764546a5-e946-4e8d-baf1-4126384304f0","Type":"ContainerStarted","Data":"d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2"}
Nov 03 19:59:57.185362 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.185347    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 19:59:57.204505 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.204452    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk" podStartSLOduration=1.004576678 podStartE2EDuration="5.204433485s" totalImagesPullingTime="4.199856807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:57.202946412 +0000 UTC m=+1189.872147883" watchObservedRunningTime="2026-11-03 19:59:57.204433485 +0000 UTC m=+1189.873634956"
Nov 03 19:59:57.711016 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.710984    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-dj76x"]
Nov 03 19:59:57.714991 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.714975    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:57.716106 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.716083    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:57.722440 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.722275    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 19:59:57.724293 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.724274    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-dj76x"]
Nov 03 19:59:57.747743 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.747712    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg778"]
Nov 03 19:59:57.751099 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.751083    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:57.762340 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.762316    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c45vx\""
Nov 03 19:59:57.763241 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.763184    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg778"]
Nov 03 19:59:57.769935 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.769910    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a267af30-6705-4247-a544-4c799fa2326b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qg778\" (UID: \"a267af30-6705-4247-a544-4c799fa2326b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:57.770092 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.770069    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzzqp\" (UniqueName: \"kubernetes.io/projected/82bd2aaa-93d7-415c-aabe-b77f54e6f3be-kube-api-access-fzzqp\") pod \"payload-processing-7bb9489676-dj76x\" (UID: \"82bd2aaa-93d7-415c-aabe-b77f54e6f3be\") " pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:57.770154 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.770123    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crkq\" (UniqueName: \"kubernetes.io/projected/a267af30-6705-4247-a544-4c799fa2326b-kube-api-access-2crkq\") pod \"payload-pre-processing-7b98bb678c-qg778\" (UID: \"a267af30-6705-4247-a544-4c799fa2326b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:57.770265 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.770246    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82bd2aaa-93d7-415c-aabe-b77f54e6f3be-plugins-config-volume\") pod \"payload-processing-7bb9489676-dj76x\" (UID: \"82bd2aaa-93d7-415c-aabe-b77f54e6f3be\") " pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:57.830264 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.830233    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 19:59:57.871400 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.871366    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82bd2aaa-93d7-415c-aabe-b77f54e6f3be-plugins-config-volume\") pod \"payload-processing-7bb9489676-dj76x\" (UID: \"82bd2aaa-93d7-415c-aabe-b77f54e6f3be\") " pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:57.871586 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.871425    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a267af30-6705-4247-a544-4c799fa2326b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qg778\" (UID: \"a267af30-6705-4247-a544-4c799fa2326b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:57.871586 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.871525    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzqp\" (UniqueName: \"kubernetes.io/projected/82bd2aaa-93d7-415c-aabe-b77f54e6f3be-kube-api-access-fzzqp\") pod \"payload-processing-7bb9489676-dj76x\" (UID: \"82bd2aaa-93d7-415c-aabe-b77f54e6f3be\") " pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:57.871586 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.871565    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crkq\" (UniqueName: \"kubernetes.io/projected/a267af30-6705-4247-a544-4c799fa2326b-kube-api-access-2crkq\") pod \"payload-pre-processing-7b98bb678c-qg778\" (UID: \"a267af30-6705-4247-a544-4c799fa2326b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:57.874110 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.872772    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82bd2aaa-93d7-415c-aabe-b77f54e6f3be-plugins-config-volume\") pod \"payload-processing-7bb9489676-dj76x\" (UID: \"82bd2aaa-93d7-415c-aabe-b77f54e6f3be\") " pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:57.874110 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.873504    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a267af30-6705-4247-a544-4c799fa2326b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qg778\" (UID: \"a267af30-6705-4247-a544-4c799fa2326b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:57.879971 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.879941    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crkq\" (UniqueName: \"kubernetes.io/projected/a267af30-6705-4247-a544-4c799fa2326b-kube-api-access-2crkq\") pod \"payload-pre-processing-7b98bb678c-qg778\" (UID: \"a267af30-6705-4247-a544-4c799fa2326b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:57.881694 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:57.881673    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzqp\" (UniqueName: \"kubernetes.io/projected/82bd2aaa-93d7-415c-aabe-b77f54e6f3be-kube-api-access-fzzqp\") pod \"payload-processing-7bb9489676-dj76x\" (UID: \"82bd2aaa-93d7-415c-aabe-b77f54e6f3be\") " pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:58.030931 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:58.030901    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 19:59:58.063255 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:58.063222    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 19:59:58.306950 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:58.306829    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-dj76x"]
Nov 03 19:59:58.349507 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:58.349474    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qg778"]
Nov 03 19:59:58.354803 ip-10-0-142-244 kubenswrapper[5282]: W1103 19:59:58.354771    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda267af30_6705_4247_a544_4c799fa2326b.slice/crio-4c7e3e3f1b452ed917e821d67b323a4165d62af17c05053e6f396e7d3fb8aac3 WatchSource:0}: Error finding container 4c7e3e3f1b452ed917e821d67b323a4165d62af17c05053e6f396e7d3fb8aac3: Status 404 returned error can't find the container with id 4c7e3e3f1b452ed917e821d67b323a4165d62af17c05053e6f396e7d3fb8aac3
Nov 03 19:59:59.200632 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:59.200592    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778" event={"ID":"a267af30-6705-4247-a544-4c799fa2326b","Type":"ContainerStarted","Data":"4c7e3e3f1b452ed917e821d67b323a4165d62af17c05053e6f396e7d3fb8aac3"}
Nov 03 19:59:59.201753 ip-10-0-142-244 kubenswrapper[5282]: I1103 19:59:59.201726    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-dj76x" event={"ID":"82bd2aaa-93d7-415c-aabe-b77f54e6f3be","Type":"ContainerStarted","Data":"dbde85e2c712202b32a3832325e25d424c6703f877ec5d6916d10da4ab36d568"}
Nov 03 20:00:01.214796 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:01.214771    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778" event={"ID":"a267af30-6705-4247-a544-4c799fa2326b","Type":"ContainerStarted","Data":"727d21d4570e2afd41939d2965c52f43bc0fdeca5648a30812d0f0c1e47f8c7d"}
Nov 03 20:00:01.215187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:01.214872    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 20:00:01.232040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:01.231994    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778" podStartSLOduration=1.687616116 podStartE2EDuration="4.231979222s" totalImagesPullingTime="2.544363106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:01.230964275 +0000 UTC m=+1193.900165744" watchObservedRunningTime="2026-11-03 20:00:01.231979222 +0000 UTC m=+1193.901180692"
Nov 03 20:00:01.613488 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:00:01.613451    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:00:02.221723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:02.221684    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-dj76x" event={"ID":"82bd2aaa-93d7-415c-aabe-b77f54e6f3be","Type":"ContainerStarted","Data":"b20f8aab6e9a1a9b26d4cb573881753f4bbf67b560563f1656e53d2dc2c0a88b"}
Nov 03 20:00:02.222226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:02.222055    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 20:00:02.237493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:02.237447    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-dj76x" podStartSLOduration=2.354268108 podStartE2EDuration="5.237410051s" totalImagesPullingTime="2.883141943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:02.236514815 +0000 UTC m=+1194.905716296" watchObservedRunningTime="2026-11-03 20:00:02.237410051 +0000 UTC m=+1194.906611559"
Nov 03 20:00:02.763097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:02.763062    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:08.195161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:08.195131    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 20:00:12.224767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:12.224733    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qg778"
Nov 03 20:00:13.228946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:13.228915    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-dj76x"
Nov 03 20:00:19.881441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:19.881397    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:23.735903 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:23.735863    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-gtfjk"]
Nov 03 20:00:23.736535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:23.736107    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk" podUID="764546a5-e946-4e8d-baf1-4126384304f0" containerName="manager" containerID="cri-o://d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2" gracePeriod=30
Nov 03 20:00:23.984438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:23.984413    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 20:00:24.106526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.106438    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764546a5-e946-4e8d-baf1-4126384304f0-kube-api-access-jgrx9\" (UniqueName: \"kubernetes.io/projected/764546a5-e946-4e8d-baf1-4126384304f0-kube-api-access-jgrx9\") pod \"764546a5-e946-4e8d-baf1-4126384304f0\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") "
Nov 03 20:00:24.106660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.106563    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764546a5-e946-4e8d-baf1-4126384304f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/764546a5-e946-4e8d-baf1-4126384304f0-metrics-certs\") pod \"764546a5-e946-4e8d-baf1-4126384304f0\" (UID: \"764546a5-e946-4e8d-baf1-4126384304f0\") "
Nov 03 20:00:24.108669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.108636    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764546a5-e946-4e8d-baf1-4126384304f0-metrics-certs" pod "764546a5-e946-4e8d-baf1-4126384304f0" (UID: "764546a5-e946-4e8d-baf1-4126384304f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:24.108811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.108672    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764546a5-e946-4e8d-baf1-4126384304f0-kube-api-access-jgrx9" pod "764546a5-e946-4e8d-baf1-4126384304f0" (UID: "764546a5-e946-4e8d-baf1-4126384304f0"). InnerVolumeSpecName "kube-api-access-jgrx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:24.208151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.208113    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgrx9\" (UniqueName: \"kubernetes.io/projected/764546a5-e946-4e8d-baf1-4126384304f0-kube-api-access-jgrx9\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:24.208151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.208145    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764546a5-e946-4e8d-baf1-4126384304f0-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:24.328854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.328821    5282 generic.go:356] "Generic (PLEG): container finished" podID="764546a5-e946-4e8d-baf1-4126384304f0" containerID="d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2" exitCode=0
Nov 03 20:00:24.329015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.328867    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk"
Nov 03 20:00:24.329015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.328891    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk" event={"ID":"764546a5-e946-4e8d-baf1-4126384304f0","Type":"ContainerDied","Data":"d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2"}
Nov 03 20:00:24.329015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.328932    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-gtfjk" event={"ID":"764546a5-e946-4e8d-baf1-4126384304f0","Type":"ContainerDied","Data":"a06beea3c4ccf4bbee4c9bc38608b7f56590d1f5da1684eda167b095bc84e41c"}
Nov 03 20:00:24.329015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.328954    5282 scope.go:122] "RemoveContainer" containerID="d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2"
Nov 03 20:00:24.338318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.338301    5282 scope.go:122] "RemoveContainer" containerID="d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2"
Nov 03 20:00:24.338570 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:00:24.338552    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2\": container with ID starting with d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2 not found: ID does not exist" containerID="d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2"
Nov 03 20:00:24.338629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.338579    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2"} err="failed to get container status \"d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2\": rpc error: code = NotFound desc = could not find container \"d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2\": container with ID starting with d67ee12162adf93d6e220e3dd743c7595272211b0b96c61f89e7efbb4394f5c2 not found: ID does not exist"
Nov 03 20:00:24.350188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.350165    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-gtfjk"]
Nov 03 20:00:24.353917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:24.353897    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-gtfjk"]
Nov 03 20:00:25.323008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:25.322973    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:25.453054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:25.453024    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764546a5-e946-4e8d-baf1-4126384304f0" path="/var/lib/kubelet/pods/764546a5-e946-4e8d-baf1-4126384304f0/volumes"
Nov 03 20:00:25.820540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:25.820499    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.407746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:27.407711    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:27.527417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:27.527388    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.875638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:28.875592    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:28.999596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:28.999565    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:31.612721 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:00:31.612683    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:00:51.304559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.304528    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:51.417183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.417147    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:51.684401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.684322    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"]
Nov 03 20:00:51.684711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.684696    5282 cpu_manager.go:414] "containerMap: removing container" podUID="764546a5-e946-4e8d-baf1-4126384304f0" containerName="manager"
Nov 03 20:00:51.684762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.684712    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764546a5-e946-4e8d-baf1-4126384304f0" containerName="manager"
Nov 03 20:00:51.684806 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.684791    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="764546a5-e946-4e8d-baf1-4126384304f0" containerName="manager"
Nov 03 20:00:51.694450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.694424    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.700700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.700675    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-11a5b0-openshift-default-dockercfg-jmpd9\""
Nov 03 20:00:51.703042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.703019    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"]
Nov 03 20:00:51.752066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752032    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752084    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752151    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752268    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752346    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752377    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hkp\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-kube-api-access-b9hkp\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752406    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752438    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0901e915-ec8e-45f3-a77c-7087560abf13-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.752607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.752520    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0901e915-ec8e-45f3-a77c-7087560abf13-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.785070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.785031    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:51.853139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853097    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0901e915-ec8e-45f3-a77c-7087560abf13-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.853342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853149    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.853342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853175    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.853342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853227    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.853342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853259    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.853342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853291    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.853722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853697    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hkp\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-kube-api-access-b9hkp\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.853889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853871    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.854009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853979    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.854153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.854007    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0901e915-ec8e-45f3-a77c-7087560abf13-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.854371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.854346    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.854525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.853723    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.854640 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.854618    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.854710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.854696    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0901e915-ec8e-45f3-a77c-7087560abf13-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.856041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.856019    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0901e915-ec8e-45f3-a77c-7087560abf13-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.856544 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.856525    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.865300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.865268    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hkp\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-kube-api-access-b9hkp\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.865989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.865966    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:51.917585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.917541    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:51.927787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.927754    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"]
Nov 03 20:00:51.932799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.932774    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:51.941384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.941322    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"]
Nov 03 20:00:51.945688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:51.945664    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b87733-openshift-default-dockercfg-lvxzx\""
Nov 03 20:00:52.008435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.008404    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:52.046296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.046271    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:52.056851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.056821    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.056868    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.056893    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.056925    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.056949    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.056983    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.057012    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wx7\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-kube-api-access-c5wx7\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.057253    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.057310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.057305    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.100705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.100675    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"]
Nov 03 20:00:52.105431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.105410    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.118379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.117070    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"]
Nov 03 20:00:52.137954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.137927    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0ed729-openshift-default-dockercfg-cflmg\""
Nov 03 20:00:52.158787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.158760    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.158925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.158794    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.158925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.158826    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.158925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.158853    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.158925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.158894    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.158925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.158921    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wx7\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-kube-api-access-c5wx7\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159001    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159100    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159191    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159227    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159292    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159330    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159361    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159480    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159499    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159562    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159640    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159665    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159667    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159691    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159690    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.159770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159710    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8rf\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-kube-api-access-4b8rf\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.159957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.159938    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.161150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.161133    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.161802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.161786    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.172455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.172427    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.177101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.177073    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wx7\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-kube-api-access-c5wx7\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.177959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.177935    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:52.252608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.252574    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:52.260302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260275    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260306    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260326    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260346    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260394    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260417    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8rf\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-kube-api-access-4b8rf\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260465    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260499    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260516    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260699    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260746    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.260940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.260899    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.261098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.261074    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.261942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.261661    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.263102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.263080    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.263252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.263180    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.272311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.272275    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.272525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.272500    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8rf\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-kube-api-access-4b8rf\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.291491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.291464    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"]
Nov 03 20:00:52.297790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.297736    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.297913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.297828    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.297913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.297872    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.317307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.315668    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:52.424793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.423810    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:52.446322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.441651    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:52.462228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.462168    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" event={"ID":"0901e915-ec8e-45f3-a77c-7087560abf13","Type":"ContainerStarted","Data":"ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0"}
Nov 03 20:00:52.462392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.462244    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" event={"ID":"0901e915-ec8e-45f3-a77c-7087560abf13","Type":"ContainerStarted","Data":"43f3270c1133af0de1b5c58404549d5ac7344d3379017319978816c1909a1061"}
Nov 03 20:00:52.543734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.542602    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podStartSLOduration=1.54258195 podStartE2EDuration="1.54258195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:52.482092672 +0000 UTC m=+1245.151294146" watchObservedRunningTime="2026-11-03 20:00:52.54258195 +0000 UTC m=+1245.211783425"
Nov 03 20:00:52.544529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.544460    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"]
Nov 03 20:00:52.548536 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:00:52.547702    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb363fe_2c71_4a71_8a21_9b366a5d42a5.slice/crio-70531b58fba1f1a1f8f96dc217d6f43dda0848bf4044da6317540f9e91036ce7 WatchSource:0}: Error finding container 70531b58fba1f1a1f8f96dc217d6f43dda0848bf4044da6317540f9e91036ce7: Status 404 returned error can't find the container with id 70531b58fba1f1a1f8f96dc217d6f43dda0848bf4044da6317540f9e91036ce7
Nov 03 20:00:52.550774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.550741    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.550844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.550828    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.550894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.550855    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.739948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.739922    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"]
Nov 03 20:00:52.742404 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:00:52.742374    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ff4d7f_c41a_4bd0_ba8a_6a92e13495e9.slice/crio-b798d6e73f022e461ca03a8555dff5c823fa58cf9a5a733ba08a8fd6e543465c WatchSource:0}: Error finding container b798d6e73f022e461ca03a8555dff5c823fa58cf9a5a733ba08a8fd6e543465c: Status 404 returned error can't find the container with id b798d6e73f022e461ca03a8555dff5c823fa58cf9a5a733ba08a8fd6e543465c
Nov 03 20:00:52.744910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.744869    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.744990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.744959    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.745032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.745000    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:00:52.763062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:52.763027    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:53.009072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.009038    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:53.014123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.014100    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:53.468694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.468651    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" event={"ID":"3eb363fe-2c71-4a71-8a21-9b366a5d42a5","Type":"ContainerStarted","Data":"a0ad6b5004ad2738b17114df3ece7175f71aca0bcd67cbccfc5e3fee6c4a5f2d"}
Nov 03 20:00:53.469190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.468714    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" event={"ID":"3eb363fe-2c71-4a71-8a21-9b366a5d42a5","Type":"ContainerStarted","Data":"70531b58fba1f1a1f8f96dc217d6f43dda0848bf4044da6317540f9e91036ce7"}
Nov 03 20:00:53.470103 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.470073    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" event={"ID":"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9","Type":"ContainerStarted","Data":"e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d"}
Nov 03 20:00:53.470213 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.470121    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" event={"ID":"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9","Type":"ContainerStarted","Data":"b798d6e73f022e461ca03a8555dff5c823fa58cf9a5a733ba08a8fd6e543465c"}
Nov 03 20:00:53.470332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.470314    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:53.471328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.471309    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:00:53.487520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.487481    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podStartSLOduration=2.487469201 podStartE2EDuration="2.487469201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:53.486282886 +0000 UTC m=+1246.155484357" watchObservedRunningTime="2026-11-03 20:00:53.487469201 +0000 UTC m=+1246.156670671"
Nov 03 20:00:53.534342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:53.534290    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podStartSLOduration=1.534274354 podStartE2EDuration="1.534274354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:53.526170521 +0000 UTC m=+1246.195372014" watchObservedRunningTime="2026-11-03 20:00:53.534274354 +0000 UTC m=+1246.203475877"
Nov 03 20:00:54.253610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.253571    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:54.258753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.258730    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:54.424435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.424400    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:54.430110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.430085    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:54.475864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.475829    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:54.476341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.475883    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:54.476877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.476855    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:00:54.476963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.476910    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:00:54.786788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.786752    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:54.910324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:54.910290    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:55.313090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:55.313055    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:55.475004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:55.474968    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:55.611464 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:55.611363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:55.800867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:55.800805    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.038055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.038021    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.212539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.212494    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.222104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.222076    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"]
Nov 03 20:00:56.225620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.225599    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.234040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.234011    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-11a5b0\""
Nov 03 20:00:56.243237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.243210    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"]
Nov 03 20:00:56.297106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.297028    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"]
Nov 03 20:00:56.300558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.300541    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.315667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.315640    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-11a5b0-dockercfg-mkcrc\""
Nov 03 20:00:56.315928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.315904    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"]
Nov 03 20:00:56.331049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.331023    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t944b\" (UniqueName: \"kubernetes.io/projected/f866abde-bff2-407c-918e-fa4aa493084c-kube-api-access-t944b\") pod \"payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.331206 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.331175    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f866abde-bff2-407c-918e-fa4aa493084c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.381474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.381432    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.433245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.432915    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t944b\" (UniqueName: \"kubernetes.io/projected/f866abde-bff2-407c-918e-fa4aa493084c-kube-api-access-t944b\") pod \"payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.433245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.433079    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f866abde-bff2-407c-918e-fa4aa493084c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.433245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.433122    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b2bd90-9bb5-469d-a8cb-194f786a1469-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.433245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.433163    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5f2w\" (UniqueName: \"kubernetes.io/projected/a7b2bd90-9bb5-469d-a8cb-194f786a1469-kube-api-access-l5f2w\") pod \"payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.437224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.434414    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f866abde-bff2-407c-918e-fa4aa493084c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.443120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.441943    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t944b\" (UniqueName: \"kubernetes.io/projected/f866abde-bff2-407c-918e-fa4aa493084c-kube-api-access-t944b\") pod \"payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.534828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.534784    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b2bd90-9bb5-469d-a8cb-194f786a1469-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.535037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.534852    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5f2w\" (UniqueName: \"kubernetes.io/projected/a7b2bd90-9bb5-469d-a8cb-194f786a1469-kube-api-access-l5f2w\") pod \"payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.535653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.535630    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b2bd90-9bb5-469d-a8cb-194f786a1469-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.539488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.539468    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:56.543610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.543586    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5f2w\" (UniqueName: \"kubernetes.io/projected/a7b2bd90-9bb5-469d-a8cb-194f786a1469-kube-api-access-l5f2w\") pod \"payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.546208 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.546164    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.613627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.613598    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:56.678509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.678450    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:56.816517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.816489    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"]
Nov 03 20:00:56.818271 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:00:56.818237    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf866abde_bff2_407c_918e_fa4aa493084c.slice/crio-d6e6cf154739f48030c889ac2b1893443761182af5f75f1b6a9a57708dfd8e9f WatchSource:0}: Error finding container d6e6cf154739f48030c889ac2b1893443761182af5f75f1b6a9a57708dfd8e9f: Status 404 returned error can't find the container with id d6e6cf154739f48030c889ac2b1893443761182af5f75f1b6a9a57708dfd8e9f
Nov 03 20:00:56.896271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.896252    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"]
Nov 03 20:00:56.898542 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:00:56.898511    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b2bd90_9bb5_469d_a8cb_194f786a1469.slice/crio-5bd8eb1b9beb7da99d439d5fb7b5ec4641829bc63ba1338e3d244f7e6c8b7313 WatchSource:0}: Error finding container 5bd8eb1b9beb7da99d439d5fb7b5ec4641829bc63ba1338e3d244f7e6c8b7313: Status 404 returned error can't find the container with id 5bd8eb1b9beb7da99d439d5fb7b5ec4641829bc63ba1338e3d244f7e6c8b7313
Nov 03 20:00:56.981957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:56.981928    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.117300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.117069    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.396098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.396022    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"]
Nov 03 20:00:57.399873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.399852    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.408228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.408190    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-11a5b0-serving-cert\""
Nov 03 20:00:57.410118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.410084    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"]
Nov 03 20:00:57.501831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.501797    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd" event={"ID":"a7b2bd90-9bb5-469d-a8cb-194f786a1469","Type":"ContainerStarted","Data":"d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d"}
Nov 03 20:00:57.502010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.501857    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd" event={"ID":"a7b2bd90-9bb5-469d-a8cb-194f786a1469","Type":"ContainerStarted","Data":"5bd8eb1b9beb7da99d439d5fb7b5ec4641829bc63ba1338e3d244f7e6c8b7313"}
Nov 03 20:00:57.502010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.501911    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:00:57.503588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.503556    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd" event={"ID":"f866abde-bff2-407c-918e-fa4aa493084c","Type":"ContainerStarted","Data":"3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611"}
Nov 03 20:00:57.503702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.503609    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:00:57.503702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.503631    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd" event={"ID":"f866abde-bff2-407c-918e-fa4aa493084c","Type":"ContainerStarted","Data":"d6e6cf154739f48030c889ac2b1893443761182af5f75f1b6a9a57708dfd8e9f"}
Nov 03 20:00:57.518368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.518325    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd" podStartSLOduration=1.518312387 podStartE2EDuration="1.518312387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:57.517167179 +0000 UTC m=+1250.186368659" watchObservedRunningTime="2026-11-03 20:00:57.518312387 +0000 UTC m=+1250.187513857"
Nov 03 20:00:57.534463 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.534334    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd" podStartSLOduration=1.534315398 podStartE2EDuration="1.534315398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:57.532890155 +0000 UTC m=+1250.202091625" watchObservedRunningTime="2026-11-03 20:00:57.534315398 +0000 UTC m=+1250.203516893"
Nov 03 20:00:57.543657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.543631    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-maas-api-tls\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.543767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.543670    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-metrics-certs\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.543881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.543849    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.544102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.544079    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsq6\" (UniqueName: \"kubernetes.io/projected/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-kube-api-access-btsq6\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.549102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.549078    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.668843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.660707    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-maas-api-tls\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.668843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.660773    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-metrics-certs\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.668843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.660846    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.668843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.661011    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsq6\" (UniqueName: \"kubernetes.io/projected/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-kube-api-access-btsq6\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.678835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.677975    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.685413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.685314    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.685413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.685349    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-maas-api-tls\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.685813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.685783    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsq6\" (UniqueName: \"kubernetes.io/projected/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-kube-api-access-btsq6\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.686090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.686072    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-metrics-certs\") pod \"maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.713075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.713041    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:57.840695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.840655    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:57.863416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.863384    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"]
Nov 03 20:00:57.868011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.867991    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:57.877408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.875364    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-5348ca-dockercfg-x698f\""
Nov 03 20:00:57.877408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.875456    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-5348ca\""
Nov 03 20:00:57.878257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.878178    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"]
Nov 03 20:00:57.976772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.976741    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae51af71-0640-4d6d-bec2-f819ef9ebad6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:57.976949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.976813    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8kb\" (UniqueName: \"kubernetes.io/projected/ae51af71-0640-4d6d-bec2-f819ef9ebad6-kube-api-access-6d8kb\") pod \"payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:57.977564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:57.977407    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:58.019218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.014299    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"]
Nov 03 20:00:58.020551 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:00:58.020522    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f2c16b_4e2a_4530_a8b8_ee032f0d4d81.slice/crio-d9ca60ef198a4e31b8fd8ff478e5b7357afb99b2491ca0a7172995ae35bc960e WatchSource:0}: Error finding container d9ca60ef198a4e31b8fd8ff478e5b7357afb99b2491ca0a7172995ae35bc960e: Status 404 returned error can't find the container with id d9ca60ef198a4e31b8fd8ff478e5b7357afb99b2491ca0a7172995ae35bc960e
Nov 03 20:00:58.078075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.078045    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae51af71-0640-4d6d-bec2-f819ef9ebad6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:58.078254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.078099    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8kb\" (UniqueName: \"kubernetes.io/projected/ae51af71-0640-4d6d-bec2-f819ef9ebad6-kube-api-access-6d8kb\") pod \"payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:58.078750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.078731    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae51af71-0640-4d6d-bec2-f819ef9ebad6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:58.089865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.089834    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8kb\" (UniqueName: \"kubernetes.io/projected/ae51af71-0640-4d6d-bec2-f819ef9ebad6-kube-api-access-6d8kb\") pod \"payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:58.184984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.184949    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:58.339308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.339273    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:58.473730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.473703    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"]
Nov 03 20:00:58.475723 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:00:58.475695    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae51af71_0640_4d6d_bec2_f819ef9ebad6.slice/crio-5608cdc109b60d69cb70e93a532553565d157f25a303cff931c8e63211eb5f08 WatchSource:0}: Error finding container 5608cdc109b60d69cb70e93a532553565d157f25a303cff931c8e63211eb5f08: Status 404 returned error can't find the container with id 5608cdc109b60d69cb70e93a532553565d157f25a303cff931c8e63211eb5f08
Nov 03 20:00:58.493904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.493880    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:58.509226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.509178    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb" event={"ID":"ae51af71-0640-4d6d-bec2-f819ef9ebad6","Type":"ContainerStarted","Data":"5608cdc109b60d69cb70e93a532553565d157f25a303cff931c8e63211eb5f08"}
Nov 03 20:00:58.510309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.510284    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m" event={"ID":"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81","Type":"ContainerStarted","Data":"d9ca60ef198a4e31b8fd8ff478e5b7357afb99b2491ca0a7172995ae35bc960e"}
Nov 03 20:00:58.960756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:58.960675    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.082879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.082847    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.358543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.358511    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.475700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.475671    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:00:59.567631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.567587    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m" event={"ID":"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81","Type":"ContainerStarted","Data":"1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003"}
Nov 03 20:00:59.571331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.568547    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:00:59.571331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.570735    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb" event={"ID":"ae51af71-0640-4d6d-bec2-f819ef9ebad6","Type":"ContainerStarted","Data":"cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5"}
Nov 03 20:00:59.571620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.571441    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:00:59.590629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.590578    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m" podStartSLOduration=2.046791865 podStartE2EDuration="2.590560835s" totalImagesPullingTime="543.76897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:59.588922273 +0000 UTC m=+1252.258123745" watchObservedRunningTime="2026-11-03 20:00:59.590560835 +0000 UTC m=+1252.259762305"
Nov 03 20:00:59.606986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.606930    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb" podStartSLOduration=2.606916227 podStartE2EDuration="2.606916227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:59.604522554 +0000 UTC m=+1252.273724023" watchObservedRunningTime="2026-11-03 20:00:59.606916227 +0000 UTC m=+1252.276117697"
Nov 03 20:00:59.975803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:00:59.975759    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:00.039784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.039749    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:00.105564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.105524    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"]
Nov 03 20:01:00.108984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.108965    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.149174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.149136    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-0ed729\""
Nov 03 20:01:00.151114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.151087    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"]
Nov 03 20:01:00.152953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.152922    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-0ed729-dockercfg-fndg4\""
Nov 03 20:01:00.179042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.179005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:00.206480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.206441    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.206678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.206499    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npd2s\" (UniqueName: \"kubernetes.io/projected/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-kube-api-access-npd2s\") pod \"payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.297969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.297932    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:00.307392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.307359    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.307530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.307415    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd2s\" (UniqueName: \"kubernetes.io/projected/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-kube-api-access-npd2s\") pod \"payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.308118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.308097    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.315143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.315113    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd2s\" (UniqueName: \"kubernetes.io/projected/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-kube-api-access-npd2s\") pod \"payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.451391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.451353    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:00.452532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.452510    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:00.495801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.495763    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"]
Nov 03 20:01:00.501315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.501112    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:00.516013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.515371    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"]
Nov 03 20:01:00.516013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.515529    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ab1e46-dockercfg-cdkjl\""
Nov 03 20:01:00.516013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.515645    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ab1e46\""
Nov 03 20:01:00.537927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.537862    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vrx\" (UniqueName: \"kubernetes.io/projected/4e978959-92f5-4c2a-afc3-91ae95b77dd9-kube-api-access-v4vrx\") pod \"payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:00.538134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.537949    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e978959-92f5-4c2a-afc3-91ae95b77dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:00.578552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.578468    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:00.641104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.638906    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vrx\" (UniqueName: \"kubernetes.io/projected/4e978959-92f5-4c2a-afc3-91ae95b77dd9-kube-api-access-v4vrx\") pod \"payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:00.641104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.638965    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e978959-92f5-4c2a-afc3-91ae95b77dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:00.641104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.640045    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e978959-92f5-4c2a-afc3-91ae95b77dd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:00.654782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.652228    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vrx\" (UniqueName: \"kubernetes.io/projected/4e978959-92f5-4c2a-afc3-91ae95b77dd9-kube-api-access-v4vrx\") pod \"payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:00.726884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.725041    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"]
Nov 03 20:01:00.829826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:00.829748    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:01.117715 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.117688    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"]
Nov 03 20:01:01.120145 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:01:01.120122    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e978959_92f5_4c2a_afc3_91ae95b77dd9.slice/crio-8ee4920e87210cc60c31d849c899b6b1c6a66cef4a83aea2bf8fde17e2b24d6d WatchSource:0}: Error finding container 8ee4920e87210cc60c31d849c899b6b1c6a66cef4a83aea2bf8fde17e2b24d6d: Status 404 returned error can't find the container with id 8ee4920e87210cc60c31d849c899b6b1c6a66cef4a83aea2bf8fde17e2b24d6d
Nov 03 20:01:01.399727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.399638    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:01.602923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.602884    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz" event={"ID":"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4","Type":"ContainerStarted","Data":"6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940"}
Nov 03 20:01:01.603091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.603012    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:01.603091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.603041    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz" event={"ID":"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4","Type":"ContainerStarted","Data":"3eb2521966e956623b22cb1b945b779a58af245b10df25feeec82331577ab6df"}
Nov 03 20:01:01.604630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.604594    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" event={"ID":"4e978959-92f5-4c2a-afc3-91ae95b77dd9","Type":"ContainerStarted","Data":"fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501"}
Nov 03 20:01:01.604747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.604651    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" event={"ID":"4e978959-92f5-4c2a-afc3-91ae95b77dd9","Type":"ContainerStarted","Data":"8ee4920e87210cc60c31d849c899b6b1c6a66cef4a83aea2bf8fde17e2b24d6d"}
Nov 03 20:01:01.623448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.623389    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz" podStartSLOduration=1.6233739740000002 podStartE2EDuration="1.623373974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:01.621779695 +0000 UTC m=+1254.290981165" watchObservedRunningTime="2026-11-03 20:01:01.623373974 +0000 UTC m=+1254.292575444"
Nov 03 20:01:01.624029 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:01:01.624000    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:01:01.638045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:01.637991    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" podStartSLOduration=1.637977818 podStartE2EDuration="1.637977818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:01.636412771 +0000 UTC m=+1254.305614242" watchObservedRunningTime="2026-11-03 20:01:01.637977818 +0000 UTC m=+1254.307179328"
Nov 03 20:01:02.139030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.138993    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"]
Nov 03 20:01:02.143027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.143002    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.147034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.147005    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-120fe3-serving-cert\""
Nov 03 20:01:02.152079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.152050    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2zz\" (UniqueName: \"kubernetes.io/projected/777432e5-8d85-41bc-b1d3-851ee8926850-kube-api-access-jt2zz\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.152271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.152097    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777432e5-8d85-41bc-b1d3-851ee8926850-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.152271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.152124    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-metrics-certs\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.152409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.152334    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.154645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.154615    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"]
Nov 03 20:01:02.222259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.222220    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:02.252885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.252844    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.253089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.252906    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2zz\" (UniqueName: \"kubernetes.io/projected/777432e5-8d85-41bc-b1d3-851ee8926850-kube-api-access-jt2zz\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.253089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.252931    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777432e5-8d85-41bc-b1d3-851ee8926850-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.253089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.252946    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-metrics-certs\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.253089 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:01:02.253028    5282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-serving-cert: secret "maas-api-e2e-worker-w0-120fe3-serving-cert" not found
Nov 03 20:01:02.253329 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:01:02.253126    5282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls podName:777432e5-8d85-41bc-b1d3-851ee8926850 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:02.753104009 +0000 UTC m=+1255.422305472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls") pod "maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh" (UID: "777432e5-8d85-41bc-b1d3-851ee8926850") : secret "maas-api-e2e-worker-w0-120fe3-serving-cert" not found
Nov 03 20:01:02.253742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.253722    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777432e5-8d85-41bc-b1d3-851ee8926850-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.255408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.255387    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-metrics-certs\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.307295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.302979    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2zz\" (UniqueName: \"kubernetes.io/projected/777432e5-8d85-41bc-b1d3-851ee8926850-kube-api-access-jt2zz\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.322754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.322714    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:02.457594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.457512    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:02.610300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.610273    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:02.768292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.768252    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:02.773276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:02.773245    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls\") pod \"maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:03.066452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:03.066373    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:03.126155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:03.126110    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:03.171919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:03.171886    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:03.246466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:03.246430    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:03.386274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:03.383368    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"]
Nov 03 20:01:03.403745 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:01:03.403705    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777432e5_8d85_41bc_b1d3_851ee8926850.slice/crio-1f0371a7d11ebb9245afc7410953a379badf2b905f2488e690a0d38caa0783ed WatchSource:0}: Error finding container 1f0371a7d11ebb9245afc7410953a379badf2b905f2488e690a0d38caa0783ed: Status 404 returned error can't find the container with id 1f0371a7d11ebb9245afc7410953a379badf2b905f2488e690a0d38caa0783ed
Nov 03 20:01:03.419465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:03.419433    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:03.620672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:03.620625    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh" event={"ID":"777432e5-8d85-41bc-b1d3-851ee8926850","Type":"ContainerStarted","Data":"1f0371a7d11ebb9245afc7410953a379badf2b905f2488e690a0d38caa0783ed"}
Nov 03 20:01:04.127986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.127809    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:04.629908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.629871    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh" event={"ID":"777432e5-8d85-41bc-b1d3-851ee8926850","Type":"ContainerStarted","Data":"4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467"}
Nov 03 20:01:04.630335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.630304    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:04.649532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.649485    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh" podStartSLOduration=2.224901177 podStartE2EDuration="2.649470813s" totalImagesPullingTime="424.569636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:04.648338264 +0000 UTC m=+1257.317539734" watchObservedRunningTime="2026-11-03 20:01:04.649470813 +0000 UTC m=+1257.318672283"
Nov 03 20:01:04.821222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.794318    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"]
Nov 03 20:01:04.821222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.812044    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:04.845248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.844466    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b42aea-serving-cert\""
Nov 03 20:01:04.860653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.859633    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"]
Nov 03 20:01:04.872625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.871629    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:04.953320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.951319    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d433013-5364-4ebe-96d9-08e53103e387-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:04.953320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.951388    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:04.953320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.951436    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-metrics-certs\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:04.953320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.951471    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qblr\" (UniqueName: \"kubernetes.io/projected/0d433013-5364-4ebe-96d9-08e53103e387-kube-api-access-7qblr\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:04.987224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:04.981684    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:05.052507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.052473    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.052651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.052528    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-metrics-certs\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.052651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.052557    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qblr\" (UniqueName: \"kubernetes.io/projected/0d433013-5364-4ebe-96d9-08e53103e387-kube-api-access-7qblr\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.052651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.052643    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d433013-5364-4ebe-96d9-08e53103e387-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.053311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.053287    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d433013-5364-4ebe-96d9-08e53103e387-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.055161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.055135    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.055326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.055305    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-metrics-certs\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.061727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.061706    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qblr\" (UniqueName: \"kubernetes.io/projected/0d433013-5364-4ebe-96d9-08e53103e387-kube-api-access-7qblr\") pod \"maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.092237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.091775    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:05.169386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.169348    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:05.529188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.513363    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"]
Nov 03 20:01:05.613824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.613747    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:05.636070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.636038    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv" event={"ID":"0d433013-5364-4ebe-96d9-08e53103e387","Type":"ContainerStarted","Data":"efbc06f65427059eac6dd1a52a70f981b7e8800c0f2333e0949435e6b923dd4b"}
Nov 03 20:01:05.737015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:05.736958    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:06.343580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:06.343496    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:06.622749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:06.622674    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:01:06.642862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:06.642829    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv" event={"ID":"0d433013-5364-4ebe-96d9-08e53103e387","Type":"ContainerStarted","Data":"1329b206448ec3d8418ccbaf21f3cc0cec363eed314564c421a8fec255d79650"}
Nov 03 20:01:06.643210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:06.642893    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:06.707299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:06.707236    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv" podStartSLOduration=2.156241155 podStartE2EDuration="2.707216767s" totalImagesPullingTime="550.975612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:06.706597867 +0000 UTC m=+1259.375799358" watchObservedRunningTime="2026-11-03 20:01:06.707216767 +0000 UTC m=+1259.376418237"
Nov 03 20:01:07.317313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:07.317278    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:08.232478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:08.232426    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:08.513588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:08.513505    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:01:08.513740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:08.513687    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:01:09.338797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:09.338770    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:09.511399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:09.511351    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:01:09.511578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:09.511420    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:10.304356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.304323    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:10.643248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.643152    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:01:10.675101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.674678    5282 generic.go:356] "Generic (PLEG): container finished" podID="0901e915-ec8e-45f3-a77c-7087560abf13" containerID="ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0" exitCode=0
Nov 03 20:01:10.675101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.674797    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" event={"ID":"0901e915-ec8e-45f3-a77c-7087560abf13","Type":"ContainerDied","Data":"ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0"}
Nov 03 20:01:10.675540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.675517    5282 scope.go:122] "RemoveContainer" containerID="ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0"
Nov 03 20:01:10.677140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.677100    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:10.677244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.677191    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:10.677290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.677257    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:10.677829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.677812    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:10.877115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:10.877077    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:11.395282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:11.395244    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:11.532221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:11.531763    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:11.635231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:11.634433    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:01:11.740310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:11.734407    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:11.744206 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:11.744168    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" event={"ID":"0901e915-ec8e-45f3-a77c-7087560abf13","Type":"ContainerStarted","Data":"d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81"}
Nov 03 20:01:12.010271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:12.009591    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:01:12.010271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:12.009720    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:01:12.010271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:12.009763    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:01:12.615222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:12.615110    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:01:12.658843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:12.658804    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:01:12.733888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:12.733839    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:13.020258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:13.009578    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:01:13.020258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:13.009637    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:01:13.414279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:13.382727    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:13.663171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:13.662412    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:01:13.916221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:13.913048    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:14.011238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.010573    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:01:14.011238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.010632    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:01:14.762104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.762068    5282 generic.go:356] "Generic (PLEG): container finished" podID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerID="e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d" exitCode=0
Nov 03 20:01:14.762540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.762140    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" event={"ID":"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9","Type":"ContainerDied","Data":"e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d"}
Nov 03 20:01:14.762584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.762546    5282 scope.go:122] "RemoveContainer" containerID="e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d"
Nov 03 20:01:14.763829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.763800    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:14.763880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.763871    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:14.763932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.763920    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:14.863977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:14.863935    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:15.057283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:15.056691    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:01:15.057283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:15.056744    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:01:15.078629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:15.078594    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:15.426441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:15.426355    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:15.572308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:15.572272    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:15.690627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:15.690544    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:15.769289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:15.769252    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" event={"ID":"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9","Type":"ContainerStarted","Data":"cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209"}
Nov 03 20:01:16.011221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.010617    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Nov 03 20:01:16.011221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.010688    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Nov 03 20:01:16.030222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.029536    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:16.243770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.243729    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:16.387938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.387813    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:16.427593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.427560    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:01:16.427941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.427918    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:01:16.428090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:16.428065    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:01:17.352560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:17.352509    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:17.427692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:17.427659    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:01:17.427985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:17.427943    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:01:17.502239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:17.501561    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:01:17.503728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:17.503414    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:01:17.505012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:17.504926    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:01:17.700127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:17.700023    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:18.250169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:18.250130    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:18.424393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:18.424354    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:01:18.424794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:18.424420    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:01:19.321429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:19.321345    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:19.425080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:19.425032    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:01:19.425612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:19.425109    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:01:19.925930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:19.925894    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.349362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:20.349330    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.425215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:20.425157    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:01:20.425215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:20.425221    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:01:20.560451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:20.560413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:20.908939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:20.908900    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.093498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.093389    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.236901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.236868    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.395526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.395487    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.425093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.425054    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 20:01:21.425281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.425129    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 20:01:21.623839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.623751    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.837335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.835908    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:21.843349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.842417    5282 generic.go:356] "Generic (PLEG): container finished" podID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerID="a0ad6b5004ad2738b17114df3ece7175f71aca0bcd67cbccfc5e3fee6c4a5f2d" exitCode=0
Nov 03 20:01:21.843349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.842588    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" event={"ID":"3eb363fe-2c71-4a71-8a21-9b366a5d42a5","Type":"ContainerDied","Data":"a0ad6b5004ad2738b17114df3ece7175f71aca0bcd67cbccfc5e3fee6c4a5f2d"}
Nov 03 20:01:21.843349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.843044    5282 scope.go:122] "RemoveContainer" containerID="a0ad6b5004ad2738b17114df3ece7175f71aca0bcd67cbccfc5e3fee6c4a5f2d"
Nov 03 20:01:21.849131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.847793    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:21.849131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.847893    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:21.849131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:21.847945    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:01:22.001974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:22.001523    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.430770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:22.430732    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:01:22.431353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:22.431331    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:01:22.432069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:22.432049    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:01:22.615409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:22.605902    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:22.850158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:22.850122    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" event={"ID":"3eb363fe-2c71-4a71-8a21-9b366a5d42a5","Type":"ContainerStarted","Data":"c693ed9062c20f06a83ff418e7f17a4cd0baeb19351551576852c47f7047b6c7"}
Nov 03 20:01:23.253270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:23.253238    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:01:23.254211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:23.254172    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:01:23.254334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:23.254247    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 20:01:23.492057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:23.491982    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:24.253087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.253052    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:01:24.253308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.253105    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 20:01:24.260736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.260701    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"]
Nov 03 20:01:24.264909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.264885    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.290687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.274436    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"simulator-w4-5348ca-kserve-self-signed-certs\""
Nov 03 20:01:24.290687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.274519    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"default-dockercfg-2h9k8\""
Nov 03 20:01:24.290687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.274635    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"kube-root-ca.crt\""
Nov 03 20:01:24.290687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.274707    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"openshift-service-ca.crt\""
Nov 03 20:01:24.304803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.304761    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"]
Nov 03 20:01:24.334338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.334306    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:24.429170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.429137    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tls-certs\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.429362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.429236    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tmp-dir\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.429362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.429257    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmnrx\" (UniqueName: \"kubernetes.io/projected/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-kube-api-access-gmnrx\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.429362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.429291    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-dshm\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.429496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.429420    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-model-cache\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.429496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.429445    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-home\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.461158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.461113    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:24.500301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.499311    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:01:24.530365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530283    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tmp-dir\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530325    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmnrx\" (UniqueName: \"kubernetes.io/projected/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-kube-api-access-gmnrx\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530366    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-dshm\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530406    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-model-cache\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530426    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-home\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530476    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tls-certs\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530686    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tmp-dir\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530835    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-model-cache\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.530896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.530870    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-home\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.532527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.532509    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-dshm\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.532854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.532836    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tls-certs\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.538524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.538501    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmnrx\" (UniqueName: \"kubernetes.io/projected/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-kube-api-access-gmnrx\") pod \"simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") " pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.618221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.613819    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:24.926511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.926479    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"]
Nov 03 20:01:24.928996 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:01:24.928960    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e487b72_dd9a_4f64_8292_fdfaf3fbacdf.slice/crio-32d5b6aa8df9817f9b64370c888d244438e016796c318f8b5d11a56f155ac1de WatchSource:0}: Error finding container 32d5b6aa8df9817f9b64370c888d244438e016796c318f8b5d11a56f155ac1de: Status 404 returned error can't find the container with id 32d5b6aa8df9817f9b64370c888d244438e016796c318f8b5d11a56f155ac1de
Nov 03 20:01:24.998297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:24.998247    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:25.253617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:25.253575    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:01:25.253787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:25.253637    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 20:01:25.400246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:25.400175    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:25.526038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:25.525945    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:25.867945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:25.867855    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" event={"ID":"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf","Type":"ContainerStarted","Data":"572f612044adffd4127ba5934144352df16b51f96e938dee6113249c379abe6b"}
Nov 03 20:01:25.867945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:25.867911    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" event={"ID":"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf","Type":"ContainerStarted","Data":"32d5b6aa8df9817f9b64370c888d244438e016796c318f8b5d11a56f155ac1de"}
Nov 03 20:01:25.912965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:25.912903    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podStartSLOduration=1.912886765 podStartE2EDuration="1.912886765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:25.911492662 +0000 UTC m=+1278.580694134" watchObservedRunningTime="2026-11-03 20:01:25.912886765 +0000 UTC m=+1278.582088235"
Nov 03 20:01:26.140044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.139957    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:26.253257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.253222    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:01:26.253448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.253278    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 20:01:26.519921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.519884    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:26.614704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.614580    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:26.699017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.698988    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:26.842497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.842408    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:26.990226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:26.986982    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:27.253684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:27.253647    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 20:01:27.253863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:27.253702    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 20:01:27.660985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:27.660902    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:28.310396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:28.310364    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:01:28.312103 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:28.312075    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:01:28.333808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:28.333772    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:29.169062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.168153    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:29.671722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.671691    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"]
Nov 03 20:01:29.675638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.675617    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.679885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.679863    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"default-dockercfg-c5bvs\""
Nov 03 20:01:29.683849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.682781    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"simulator-w0-120fe3-kserve-self-signed-certs\""
Nov 03 20:01:29.687986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.685828    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"kube-root-ca.crt\""
Nov 03 20:01:29.687986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.686298    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"openshift-service-ca.crt\""
Nov 03 20:01:29.687986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.687932    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"]
Nov 03 20:01:29.763084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.763051    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:29.793233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.793185    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tmp-dir\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.793403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.793322    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-model-cache\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.793403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.793365    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zj22\" (UniqueName: \"kubernetes.io/projected/f873dc7e-faaa-4b5d-8b32-298726b5aef8-kube-api-access-2zj22\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.793403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.793388    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-dshm\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.793507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.793455    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tls-certs\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.793507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.793494    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-home\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.894310    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-dshm\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.894391    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tls-certs\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.894436    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-home\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.894510    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tmp-dir\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.894621    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-model-cache\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.894683    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zj22\" (UniqueName: \"kubernetes.io/projected/f873dc7e-faaa-4b5d-8b32-298726b5aef8-kube-api-access-2zj22\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.895886    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-home\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.896876    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tmp-dir\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.898221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.897147    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-model-cache\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.906543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.906506    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-dshm\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.907755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.907727    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tls-certs\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.910772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.910691    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zj22\" (UniqueName: \"kubernetes.io/projected/f873dc7e-faaa-4b5d-8b32-298726b5aef8-kube-api-access-2zj22\") pod \"simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") " pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:29.991579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:29.991541    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:30.196982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:30.196946    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:30.265259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:30.265232    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"]
Nov 03 20:01:30.267464 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:01:30.267430    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf873dc7e_faaa_4b5d_8b32_298726b5aef8.slice/crio-86fc2258952368049472257fd559021470efbbcc97668c9c71fca548edae330a WatchSource:0}: Error finding container 86fc2258952368049472257fd559021470efbbcc97668c9c71fca548edae330a: Status 404 returned error can't find the container with id 86fc2258952368049472257fd559021470efbbcc97668c9c71fca548edae330a
Nov 03 20:01:30.900716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:30.900674    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" event={"ID":"f873dc7e-faaa-4b5d-8b32-298726b5aef8","Type":"ContainerStarted","Data":"989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b"}
Nov 03 20:01:30.900895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:30.900728    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" event={"ID":"f873dc7e-faaa-4b5d-8b32-298726b5aef8","Type":"ContainerStarted","Data":"86fc2258952368049472257fd559021470efbbcc97668c9c71fca548edae330a"}
Nov 03 20:01:30.917687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:30.917622    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podStartSLOduration=1.917605015 podStartE2EDuration="1.917605015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:30.916869947 +0000 UTC m=+1283.586071417" watchObservedRunningTime="2026-11-03 20:01:30.917605015 +0000 UTC m=+1283.586806486"
Nov 03 20:01:30.945404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:30.945368    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:30.991673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:30.991641    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:31.614698 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:01:31.614661    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:01:31.661279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:31.661246    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:31.785267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:31.785228    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:32.436233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:32.433853    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:32.801724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:32.801687    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:33.133571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:33.133484    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:33.943796 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:33.943735    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:34.614559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:34.614525    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:34.627367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:34.627340    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:34.703386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:34.703350    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:34.931929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:34.931850    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:01:35.430347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:35.429095    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:36.115883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:36.115823    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:36.903015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:36.902984    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.128880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:37.128843    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:37.703943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:37.703910    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:38.392023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:38.391983    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.133132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:39.131694    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.749685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:39.749648    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:39.991913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:39.991877    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:40.004390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:40.004330    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:40.390534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:40.390451    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:40.965324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:40.965298    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:01:41.213657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.213618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:41.238371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.238339    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"]
Nov 03 20:01:41.244429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.244400    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.253825 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.253796    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"]
Nov 03 20:01:41.254239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.254217    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"kube-root-ca.crt\""
Nov 03 20:01:41.254384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.254369    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"default-dockercfg-5t6p5\""
Nov 03 20:01:41.254455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.254444    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"openshift-service-ca.crt\""
Nov 03 20:01:41.254535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.254473    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"premium-w5-ab1e46-kserve-self-signed-certs\""
Nov 03 20:01:41.335156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.335090    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:41.387223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.380539    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tmp-dir\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.387223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.380598    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-home\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.387223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.380679    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-model-cache\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.387223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.380734    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-dshm\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.387223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.380838    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tls-certs\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.387223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.380861    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7kv\" (UniqueName: \"kubernetes.io/projected/e40ae433-7695-44d1-ba58-0614ce2aa2f5-kube-api-access-np7kv\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.487951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.481483    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-dshm\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.487951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.481621    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tls-certs\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.487951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.481651    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7kv\" (UniqueName: \"kubernetes.io/projected/e40ae433-7695-44d1-ba58-0614ce2aa2f5-kube-api-access-np7kv\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.487951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.481712    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tmp-dir\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.487951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.481745    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-home\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.487951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.481819    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-model-cache\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.487951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.485622    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-dshm\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.489973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.488839    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tls-certs\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.489973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.489415    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tmp-dir\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.489973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.489660    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-home\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.489973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.489900    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-model-cache\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.499487 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.498661    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7kv\" (UniqueName: \"kubernetes.io/projected/e40ae433-7695-44d1-ba58-0614ce2aa2f5-kube-api-access-np7kv\") pod \"premium-w5-ab1e46-kserve-7b44bfff68-hjx2g\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") " pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.568914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.568872    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:41.874584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.874556    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"]
Nov 03 20:01:41.876988 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:01:41.876960    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode40ae433_7695_44d1_ba58_0614ce2aa2f5.slice/crio-80ed36d14c31e651d221cd0d2d81fc44cdae297fb1f03fcd7c18119f3d7f5b70 WatchSource:0}: Error finding container 80ed36d14c31e651d221cd0d2d81fc44cdae297fb1f03fcd7c18119f3d7f5b70: Status 404 returned error can't find the container with id 80ed36d14c31e651d221cd0d2d81fc44cdae297fb1f03fcd7c18119f3d7f5b70
Nov 03 20:01:41.894379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.894352    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:41.962149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:41.962118    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" event={"ID":"e40ae433-7695-44d1-ba58-0614ce2aa2f5","Type":"ContainerStarted","Data":"80ed36d14c31e651d221cd0d2d81fc44cdae297fb1f03fcd7c18119f3d7f5b70"}
Nov 03 20:01:42.684399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:42.684364    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:42.969752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:42.969673    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" event={"ID":"e40ae433-7695-44d1-ba58-0614ce2aa2f5","Type":"ContainerStarted","Data":"482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620"}
Nov 03 20:01:42.986503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:42.986319    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podStartSLOduration=1.986303937 podStartE2EDuration="1.986303937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:42.985514121 +0000 UTC m=+1295.654715592" watchObservedRunningTime="2026-11-03 20:01:42.986303937 +0000 UTC m=+1295.655505409"
Nov 03 20:01:43.384021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:43.383975    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:43.569597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:43.569564    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:44.175475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:44.175442    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:44.862488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:44.862454    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:45.481081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:45.481039    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:46.037655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:46.037617    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:46.160866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:46.160817    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:46.888021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:46.887979    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:47.507417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:47.507381    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:48.087139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:48.087100    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:48.218415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:48.218380    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:49.127184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:49.127151    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:49.781153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:49.781113    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:50.508926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:50.508878    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:51.208412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:51.208351    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:51.570052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:51.570023    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:51.582647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:51.582621    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:51.961723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:51.961641    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"]
Nov 03 20:01:51.967023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:51.966993    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:51.976154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:51.976132    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"]
Nov 03 20:01:51.979873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:51.979840    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"premium-w4-5348ca-kserve-self-signed-certs\""
Nov 03 20:01:52.009327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.009300    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:52.026647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.026620    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:01:52.108320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.108287    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-model-cache\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.108498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.108434    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tls-certs\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.108498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.108482    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-home\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.108635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.108507    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tmp-dir\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.108635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.108530    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-dshm\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.108635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.108574    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7sfb\" (UniqueName: \"kubernetes.io/projected/c8a5328f-7423-4d5f-8b53-eb8a10411b75-kube-api-access-r7sfb\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.124823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.124800    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:52.210046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210004    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-model-cache\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210066    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tls-certs\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210115    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-home\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210144    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tmp-dir\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210175    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-dshm\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210213    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7sfb\" (UniqueName: \"kubernetes.io/projected/c8a5328f-7423-4d5f-8b53-eb8a10411b75-kube-api-access-r7sfb\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210470    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-model-cache\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210583    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tmp-dir\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.210735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.210712    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-home\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.212464 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.212414    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-dshm\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.212738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.212720    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tls-certs\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.217809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.217784    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7sfb\" (UniqueName: \"kubernetes.io/projected/c8a5328f-7423-4d5f-8b53-eb8a10411b75-kube-api-access-r7sfb\") pod \"premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") " pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.283142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.283108    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:52.557583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.557558    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"]
Nov 03 20:01:52.559873 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:01:52.559842    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a5328f_7423_4d5f_8b53_eb8a10411b75.slice/crio-b0957725e70e5d205a5c7c47d34d14f7b4dddc9825af38d40ba9e938daf2b355 WatchSource:0}: Error finding container b0957725e70e5d205a5c7c47d34d14f7b4dddc9825af38d40ba9e938daf2b355: Status 404 returned error can't find the container with id b0957725e70e5d205a5c7c47d34d14f7b4dddc9825af38d40ba9e938daf2b355
Nov 03 20:01:52.691276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:52.691243    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:53.022171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:53.022132    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" event={"ID":"c8a5328f-7423-4d5f-8b53-eb8a10411b75","Type":"ContainerStarted","Data":"b255f3032123e2f76ee13a4cf97b787e46204609b2696f0faa0b0fb08640d8b6"}
Nov 03 20:01:53.022364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:53.022187    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" event={"ID":"c8a5328f-7423-4d5f-8b53-eb8a10411b75","Type":"ContainerStarted","Data":"b0957725e70e5d205a5c7c47d34d14f7b4dddc9825af38d40ba9e938daf2b355"}
Nov 03 20:01:53.038107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:53.038055    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podStartSLOduration=2.038039212 podStartE2EDuration="2.038039212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:53.03728366 +0000 UTC m=+1305.706485130" watchObservedRunningTime="2026-11-03 20:01:53.038039212 +0000 UTC m=+1305.707240683"
Nov 03 20:01:53.283668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:53.283584    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:01:53.405589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:53.405561    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:54.089830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:54.089799    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:54.416285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:54.416155    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:55.095528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:55.095498    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:55.837222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:55.829495    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:56.587081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:56.587053    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:57.222532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:57.222500    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:57.488486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:57.488449    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:57.985449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:57.985418    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:58.694076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:58.694041    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:01:59.185773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:59.185728    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:01:59.410385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:01:59.410346    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:00.473467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:00.473436    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:01.385373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:01.385328    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:01.614609 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:02:01.614575    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:02:02.009529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.009499    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:02.058918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.058881    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"]
Nov 03 20:02:02.063641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.063619    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.070938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.070910    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"]
Nov 03 20:02:02.071387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.071363    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-120fe3\"/\"premium-w0-120fe3-kserve-self-signed-certs\""
Nov 03 20:02:02.135849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.135821    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:02.205396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.202671    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-home\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.205396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.202742    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fmc\" (UniqueName: \"kubernetes.io/projected/68d5d678-8552-4d54-a140-210d27813c3d-kube-api-access-l8fmc\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.205396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.202794    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-tmp-dir\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.205396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.202822    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d5d678-8552-4d54-a140-210d27813c3d-tls-certs\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.205396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.202875    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-dshm\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.205396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.202916    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-model-cache\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.284290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.284184    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:02:02.296807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.296779    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:02:02.304612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.304586    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-tmp-dir\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.304736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.304616    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d5d678-8552-4d54-a140-210d27813c3d-tls-certs\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.304736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.304645    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-dshm\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.304821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.304748    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-model-cache\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.304905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.304889    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-home\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.304961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.304935    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fmc\" (UniqueName: \"kubernetes.io/projected/68d5d678-8552-4d54-a140-210d27813c3d-kube-api-access-l8fmc\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.305132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.305115    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-tmp-dir\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.305266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.305186    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-model-cache\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.305266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.305231    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-home\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.306895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.306873    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-dshm\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.307330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.307311    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d5d678-8552-4d54-a140-210d27813c3d-tls-certs\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.312502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.312479    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fmc\" (UniqueName: \"kubernetes.io/projected/68d5d678-8552-4d54-a140-210d27813c3d-kube-api-access-l8fmc\") pod \"premium-w0-120fe3-kserve-6d54b95c8f-9jh95\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") " pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.381052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.381015    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:02.655003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.654972    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"]
Nov 03 20:02:02.657217 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:02:02.657177    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68d5d678_8552_4d54_a140_210d27813c3d.slice/crio-38ff8ce6b35e19b910033e87610d0e1a88dc20743f347e543bb6c58beefb37fe WatchSource:0}: Error finding container 38ff8ce6b35e19b910033e87610d0e1a88dc20743f347e543bb6c58beefb37fe: Status 404 returned error can't find the container with id 38ff8ce6b35e19b910033e87610d0e1a88dc20743f347e543bb6c58beefb37fe
Nov 03 20:02:02.698761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:02.698707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:03.106489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:03.106446    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" event={"ID":"68d5d678-8552-4d54-a140-210d27813c3d","Type":"ContainerStarted","Data":"9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6"}
Nov 03 20:02:03.106694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:03.106498    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" event={"ID":"68d5d678-8552-4d54-a140-210d27813c3d","Type":"ContainerStarted","Data":"38ff8ce6b35e19b910033e87610d0e1a88dc20743f347e543bb6c58beefb37fe"}
Nov 03 20:02:03.117889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:03.117865    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:02:03.124882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:03.124836    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podStartSLOduration=1.1248242529999999 podStartE2EDuration="1.124824253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:03.123103439 +0000 UTC m=+1315.792304908" watchObservedRunningTime="2026-11-03 20:02:03.124824253 +0000 UTC m=+1315.794025723"
Nov 03 20:02:03.381146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:03.381062    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:03.489257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:03.489223    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:04.194189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:04.194149    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:04.314051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:04.314001    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:05.004046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:05.004007    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:05.706471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:05.706440    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:06.398337 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:06.398289    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:07.086808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:07.086766    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:08.012039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:08.011947    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:08.924773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:08.924732    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:09.750932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:09.750904    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:10.714987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:10.714954    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:11.509382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:11.509323    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:12.209904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:12.209870    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:12.381347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:12.381309    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:12.394116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:12.394090    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:12.975903 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:12.975873    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:13.182120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:13.182091    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:02:13.791447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:13.791411    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:14.504260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:14.504189    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:15.398146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:15.398115    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:16.094629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:16.094592    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:16.807931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:16.807896    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:17.489799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:17.489756    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:18.193179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:18.193144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:18.990397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:18.990361    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:19.485658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:19.485629    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:02:19.617320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:19.617274    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:19.986231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:19.986179    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:20.541580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:20.541535    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:21.532040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:21.532006    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:22.001286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.001246    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"]
Nov 03 20:02:22.008339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.008314    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.018123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.018090    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"]
Nov 03 20:02:22.033015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.032958    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-5348ca-openshift-default-dockercfg-jql62\""
Nov 03 20:02:22.069053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.069023    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.069314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.069086    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.069314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.069183    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aff8592-b84b-4bd4-8806-c748cf120d9e-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.069314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.069271    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.069873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.069308    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.069992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.069914    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.070069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.070007    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.070142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.070092    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2cs\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-kube-api-access-np2cs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.070142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.070129    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.170702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170668    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.170702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170705    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.170952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170728    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.170952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170779    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.170952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170845    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2cs\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-kube-api-access-np2cs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.170952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170873    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.170952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170944    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.171245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.170994    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.171245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.171026    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aff8592-b84b-4bd4-8806-c748cf120d9e-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.171365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.171240    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.171420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.171374    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.171460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.171436    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.171598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.171575    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.172445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.172416    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aff8592-b84b-4bd4-8806-c748cf120d9e-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.173542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.173511    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.173662 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.173629    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.179257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.179228    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2cs\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-kube-api-access-np2cs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.179515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.179492    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.325408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.325319    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:22.326755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.326706    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:22.645213 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.645166    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"]
Nov 03 20:02:22.647160 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:02:22.647133    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aff8592_b84b_4bd4_8806_c748cf120d9e.slice/crio-967646f2a874323921ed9d64eed48478127fab06bdb3602613cdbfb9af29e1e2 WatchSource:0}: Error finding container 967646f2a874323921ed9d64eed48478127fab06bdb3602613cdbfb9af29e1e2: Status 404 returned error can't find the container with id 967646f2a874323921ed9d64eed48478127fab06bdb3602613cdbfb9af29e1e2
Nov 03 20:02:22.649759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.649722    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:02:22.649844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.649795    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:02:22.649844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:22.649835    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:02:23.231271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:23.231233    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" event={"ID":"8aff8592-b84b-4bd4-8806-c748cf120d9e","Type":"ContainerStarted","Data":"9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c"}
Nov 03 20:02:23.231752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:23.231289    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" event={"ID":"8aff8592-b84b-4bd4-8806-c748cf120d9e","Type":"ContainerStarted","Data":"967646f2a874323921ed9d64eed48478127fab06bdb3602613cdbfb9af29e1e2"}
Nov 03 20:02:23.240942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:23.240912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:23.282512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:23.282450    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podStartSLOduration=2.282426384 podStartE2EDuration="2.282426384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:23.252722318 +0000 UTC m=+1335.921923789" watchObservedRunningTime="2026-11-03 20:02:23.282426384 +0000 UTC m=+1335.951627856"
Nov 03 20:02:23.327972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:23.327940    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:23.329278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:23.329253    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:02:23.329409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:23.329301    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:02:24.044055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:24.044019    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:24.327714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:24.327638    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:02:24.327714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:24.327685    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:02:24.898867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:24.898828    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:25.328055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:25.328015    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:02:25.328514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:25.328072    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:02:25.611221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:25.611111    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:26.309177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:26.309142    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:26.335636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:26.335598    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:02:26.337589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:26.337555    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:02:27.009046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:27.009014    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:27.328243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:27.328136    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:02:27.328243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:27.328233    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:02:27.808878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:27.808848    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:28.327571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:28.327537    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:02:28.327735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:28.327591    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 03 20:02:28.502688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:28.502657    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:29.209671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:29.209635    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:29.334644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:29.334611    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:29.335020 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:29.334993    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:29.336179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:29.336161    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:02:30.026684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:30.026651    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:30.691508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:30.691468    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:31.434373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:31.434341    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:31.614184 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:02:31.614154    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:02:32.095548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:32.095505    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:32.831223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:32.818400    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:33.607393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:33.607363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:34.483791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:34.483709    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:35.104106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:35.104060    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:35.807875 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:35.807828    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:36.546245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:36.546212    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:37.663567 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:37.663538    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:38.820254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:38.820188    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:41.980354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:41.980311    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:42.092995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:42.092942    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:42.695946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:42.695919    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:43.607740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:43.607691    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:44.406449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:44.406419    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:45.107014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:45.106921    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:45.486505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:45.486471    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:02:45.797040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:45.796940    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:46.716139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:46.716090    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:47.414485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:47.414440    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:48.408682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:48.408654    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:49.207385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:49.207306    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:50.010853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:50.010816    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:50.834129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:50.834098    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:51.879010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:51.878980    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:52.608656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:52.608625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:53.297780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:53.297745    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:53.597102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:53.597019    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:54.054989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:54.054953    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:54.810932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:54.810901    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:55.508466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:55.508434    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.216595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:56.216557    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:56.908407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:56.908373    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:57.894984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:57.894950    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:58.818994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:58.818957    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:02:59.509136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:02:59.509039    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:00.187826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:00.187791    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:00.885601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:00.885574    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:01.615447 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:03:01.615409    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:03:01.792856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:01.792829    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:02.712455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:02.712411    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.646080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:03.646051    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:03.927859 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:03.927782    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"]
Nov 03 20:03:03.932238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:03.932219    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:03.936712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:03.936687    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:03:03.936852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:03.936753    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:03:03.937805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:03.937779    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"]
Nov 03 20:03:04.014423 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.014384    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xzc\" (UniqueName: \"kubernetes.io/projected/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-kube-api-access-d2xzc\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.014592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.014491    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.014592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.014579    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.115053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.115020    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.115274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.115091    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.115274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.115141    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xzc\" (UniqueName: \"kubernetes.io/projected/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-kube-api-access-d2xzc\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.115897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.115869    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.117510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.117482    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.122854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.122831    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xzc\" (UniqueName: \"kubernetes.io/projected/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-kube-api-access-d2xzc\") pod \"maas-api-revoke-sub-0c823108-g5hh6\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.248243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.248191    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:04.290641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.290613    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:04.520611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.520583    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"]
Nov 03 20:03:04.522567 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:04.522544    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2e2a4d_88c9_4af1_9bb1_3b3bbd3918b8.slice/crio-87c9f63c83d9acdcc0e2003791025b2fde20bcadd4646656896d24df964eb975 WatchSource:0}: Error finding container 87c9f63c83d9acdcc0e2003791025b2fde20bcadd4646656896d24df964eb975: Status 404 returned error can't find the container with id 87c9f63c83d9acdcc0e2003791025b2fde20bcadd4646656896d24df964eb975
Nov 03 20:03:04.524269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:04.524251    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:03:05.000797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:05.000768    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:05.475471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:05.475391    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6" event={"ID":"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8","Type":"ContainerStarted","Data":"87c9f63c83d9acdcc0e2003791025b2fde20bcadd4646656896d24df964eb975"}
Nov 03 20:03:05.797085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:05.797050    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:06.717379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:06.717349    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:07.392049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.392014    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"]
Nov 03 20:03:07.397375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.397353    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.405301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.405278    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"]
Nov 03 20:03:07.406391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.405962    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ab1e46-openshift-default-dockercfg-v2sht\""
Nov 03 20:03:07.412481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.412455    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:07.475466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.475385    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.479995    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.480054    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.480095    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4fr\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-kube-api-access-lk4fr\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.480127    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.480164    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.480239    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.480276    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.480555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.480413    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581031    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581077    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581177    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4fr\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-kube-api-access-lk4fr\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581212    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581241    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581275    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581325    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581449    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581475    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581494    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581587    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581637    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.581829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.581809    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.582302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.582274    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.583790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.583763    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.583878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.583821    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.589662 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.589636    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.589751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.589738    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4fr\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-kube-api-access-lk4fr\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.716438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.716361    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ab1e46-openshift-default-dockercfg-v2sht\""
Nov 03 20:03:07.724059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.724035    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:07.878442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:07.878413    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:03:08.003946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.003921    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"]
Nov 03 20:03:08.005679 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:08.005657    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf008b1c7_64d5_4ed3_8954_b789905c0f0e.slice/crio-cf0e660d06df7761ca479188091db16d010301871ed853289cc34834bf5763a5 WatchSource:0}: Error finding container cf0e660d06df7761ca479188091db16d010301871ed853289cc34834bf5763a5: Status 404 returned error can't find the container with id cf0e660d06df7761ca479188091db16d010301871ed853289cc34834bf5763a5
Nov 03 20:03:08.008366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.008334    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:08.008450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.008393    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:08.008450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.008423    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:08.108941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.108903    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:08.510355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.506914    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" event={"ID":"f008b1c7-64d5-4ed3-8954-b789905c0f0e","Type":"ContainerStarted","Data":"dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20"}
Nov 03 20:03:08.510355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.506989    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" event={"ID":"f008b1c7-64d5-4ed3-8954-b789905c0f0e","Type":"ContainerStarted","Data":"cf0e660d06df7761ca479188091db16d010301871ed853289cc34834bf5763a5"}
Nov 03 20:03:08.548225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.536034    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podStartSLOduration=1.536011798 podStartE2EDuration="1.536011798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:08.53074941 +0000 UTC m=+1381.199950882" watchObservedRunningTime="2026-11-03 20:03:08.536011798 +0000 UTC m=+1381.205213270"
Nov 03 20:03:08.724332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:08.724283    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:09.208272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:09.206974    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:09.730223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:09.729041    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:03:09.730223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:09.729108    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:03:10.114109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:10.113980    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.731224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:10.727114    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:03:10.731224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:10.727226    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:03:11.158305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:11.148911    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:11.404438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:11.404388    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:11.405167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:11.405131    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.728228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:11.727611    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:03:11.728228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:11.727711    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:03:12.113274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:12.111669    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.470288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:12.467255    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:12.724538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:12.724453    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:03:12.724538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:12.724518    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:03:13.101911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:13.101887    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:13.724562 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:13.724518    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:03:13.724981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:13.724577    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:03:13.999261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:13.999181    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:14.617860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:14.617823    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6" event={"ID":"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8","Type":"ContainerStarted","Data":"b05588c98e2e20f6baf1af24af7f84c2230c91e1d1a5568b965ac83cb6408431"}
Nov 03 20:03:14.634254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:14.634189    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6" podStartSLOduration=2.036294501 podStartE2EDuration="11.634174899s" totalImagesPullingTime="9.597880398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:14.632255677 +0000 UTC m=+1387.301457147" watchObservedRunningTime="2026-11-03 20:03:14.634174899 +0000 UTC m=+1387.303376366"
Nov 03 20:03:14.724932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:14.724845    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 20:03:14.724932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:14.724898    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 20:03:14.916097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:14.915995    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:15.624219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:15.624172    5282 generic.go:356] "Generic (PLEG): container finished" podID="6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" containerID="b05588c98e2e20f6baf1af24af7f84c2230c91e1d1a5568b965ac83cb6408431" exitCode=0
Nov 03 20:03:15.624434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:15.624242    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6" event={"ID":"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8","Type":"ContainerDied","Data":"b05588c98e2e20f6baf1af24af7f84c2230c91e1d1a5568b965ac83cb6408431"}
Nov 03 20:03:15.729335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:15.729290    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:15.729727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:15.729640    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:15.730519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:15.730490    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:03:15.900459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:15.900353    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:16.699914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.699866    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:16.803307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.803286    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:16.848258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.848225    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-kube-api-access-d2xzc\" (UniqueName: \"kubernetes.io/projected/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-kube-api-access-d2xzc\") pod \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") "
Nov 03 20:03:16.848428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.848296    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-xks-ca\") pod \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") "
Nov 03 20:03:16.848428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.848408    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-openshift-ca\") pod \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\" (UID: \"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8\") "
Nov 03 20:03:16.848767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.848740    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-openshift-ca" pod "6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" (UID: "6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:16.848889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.848809    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.851088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.851064    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-kube-api-access-d2xzc" pod "6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" (UID: "6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8"). InnerVolumeSpecName "kube-api-access-d2xzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:16.851088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.851064    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-xks-ca" pod "6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" (UID: "6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:16.950361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.950281    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xzc\" (UniqueName: \"kubernetes.io/projected/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-kube-api-access-d2xzc\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:16.950361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:16.950314    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:17.268529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.268493    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"]
Nov 03 20:03:17.268998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.268981    5282 cpu_manager.go:414] "containerMap: removing container" podUID="6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" containerName="revoke-keys"
Nov 03 20:03:17.268998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.268996    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" containerName="revoke-keys"
Nov 03 20:03:17.269149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.269101    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" containerName="revoke-keys"
Nov 03 20:03:17.272427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.272407    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.277217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.277176    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"distinct-w3-b87733-kserve-self-signed-certs\""
Nov 03 20:03:17.277344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.277214    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"openshift-service-ca.crt\""
Nov 03 20:03:17.277344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.277285    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"kube-root-ca.crt\""
Nov 03 20:03:17.277344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.277295    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"default-dockercfg-9qptp\""
Nov 03 20:03:17.285368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.285343    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"]
Nov 03 20:03:17.348851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.348819    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:17.354064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.354042    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-dshm\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.354175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.354089    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-home\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.354175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.354114    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tls-certs\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.354270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.354219    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-model-cache\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.354270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.354243    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7jw\" (UniqueName: \"kubernetes.io/projected/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-kube-api-access-zf7jw\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.354429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.354409    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tmp-dir\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.484337 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.484281    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-home\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.484538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.484367    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tls-certs\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.484538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.484474    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-model-cache\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.484538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.484503    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7jw\" (UniqueName: \"kubernetes.io/projected/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-kube-api-access-zf7jw\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.484741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.484706    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tmp-dir\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.484824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.484809    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-dshm\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.485502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.485479    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-home\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.486479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.486457    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tmp-dir\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.488924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.488892    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tls-certs\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.489039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.488879    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-model-cache\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.489682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.489662    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-dshm\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.494289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.494255    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7jw\" (UniqueName: \"kubernetes.io/projected/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-kube-api-access-zf7jw\") pod \"distinct-w3-b87733-kserve-658b44f948-bs9hw\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.592657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.592505    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:17.638401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.638374    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"
Nov 03 20:03:17.638586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.638403    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6" event={"ID":"6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8","Type":"ContainerDied","Data":"87c9f63c83d9acdcc0e2003791025b2fde20bcadd4646656896d24df964eb975"}
Nov 03 20:03:17.638586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.638451    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c9f63c83d9acdcc0e2003791025b2fde20bcadd4646656896d24df964eb975"
Nov 03 20:03:17.648584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.648552    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:17.869612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:17.869584    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"]
Nov 03 20:03:17.871102 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:17.871077    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57ffb279_2c83_4ad1_aa9e_f3b1ed46fe9d.slice/crio-dd9168a20399197e1a38abd56b2ebd92ac763ebe286e3abc2783c9087c66b5c9 WatchSource:0}: Error finding container dd9168a20399197e1a38abd56b2ebd92ac763ebe286e3abc2783c9087c66b5c9: Status 404 returned error can't find the container with id dd9168a20399197e1a38abd56b2ebd92ac763ebe286e3abc2783c9087c66b5c9
Nov 03 20:03:18.317577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:18.317545    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:18.645483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:18.645401    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" event={"ID":"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d","Type":"ContainerStarted","Data":"36cb5ac553e0182881017cd7c6ec7be4d1cd386ee2cf0c4234a5287a4ce3538a"}
Nov 03 20:03:18.645483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:18.645447    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" event={"ID":"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d","Type":"ContainerStarted","Data":"dd9168a20399197e1a38abd56b2ebd92ac763ebe286e3abc2783c9087c66b5c9"}
Nov 03 20:03:18.666572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:18.666473    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podStartSLOduration=1.666459606 podStartE2EDuration="1.666459606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:18.664792351 +0000 UTC m=+1391.333993833" watchObservedRunningTime="2026-11-03 20:03:18.666459606 +0000 UTC m=+1391.335661075"
Nov 03 20:03:19.118730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:19.118694    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:19.592791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:19.592759    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:19.925929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:19.925851    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:20.052842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:20.052809    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:20.616632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:20.616591    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.421447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:21.421414    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:21.988650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:21.988619    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"]
Nov 03 20:03:21.992575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:21.992558    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.001473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.001218    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b42aea-openshift-default-dockercfg-7bhzv\""
Nov 03 20:03:22.012385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.012357    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"]
Nov 03 20:03:22.184630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.184592    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sspj\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-kube-api-access-6sspj\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.184818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.184639    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.184818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.184662    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.184818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.184680    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.184991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.184905    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.184991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.184953    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.185091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.185017    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.185091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.185070    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.185287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.185268    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.241096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.240219    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"]
Nov 03 20:03:22.244690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.244660    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.254650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.254630    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-120fe3-openshift-default-dockercfg-7jshr\""
Nov 03 20:03:22.256263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.256238    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"]
Nov 03 20:03:22.284883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.284851    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:22.330071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.330029    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sspj\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-kube-api-access-6sspj\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.330354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.330325    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.330519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.330501    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.330638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.330620    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.330829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.330811    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.330950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.330936    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.331067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.331051    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.331185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.331170    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.331378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.331360    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.332527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.332499    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.337917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.333938    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.338061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.334622    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.338061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.335119    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.338061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.335876    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.344258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.344228    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sspj\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-kube-api-access-6sspj\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.347031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.346989    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.348503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.348478    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.351985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.351964    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.432074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432032    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432089    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tb7\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-kube-api-access-l7tb7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432244    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432293    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432376    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432420    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a96be7-05f9-4135-be1d-9ab1bb17948a-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432488    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432599    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.432646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.432633    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533543    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533604    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tb7\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-kube-api-access-l7tb7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533634    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533656    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533676    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533694    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a96be7-05f9-4135-be1d-9ab1bb17948a-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533712    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533777    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.533824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533805    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.534215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.533999    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.534215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.534013    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.534329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.534218    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.534526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.534498    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.534759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.534739    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a96be7-05f9-4135-be1d-9ab1bb17948a-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.535953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.535930    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.536250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.536234    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.543858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.543838    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.544243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.544187    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tb7\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-kube-api-access-l7tb7\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.561994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.561971    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:22.608955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.608919    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:22.838859 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.838824    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"]
Nov 03 20:03:22.840805 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:22.840779    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a96be7_05f9_4135_be1d_9ab1bb17948a.slice/crio-237f965007ff9f2097955ea8433a7c7841ce2965ec178ae386cf683ffddf6888 WatchSource:0}: Error finding container 237f965007ff9f2097955ea8433a7c7841ce2965ec178ae386cf683ffddf6888: Status 404 returned error can't find the container with id 237f965007ff9f2097955ea8433a7c7841ce2965ec178ae386cf683ffddf6888
Nov 03 20:03:22.844577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.844539    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:22.844734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.844614    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:22.844734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.844643    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:22.902403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.902379    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"]
Nov 03 20:03:22.903853 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:22.903828    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd7bc8d_1d27_4642_a1fb_676c08dae7bc.slice/crio-6ee709430472e7662db34382f38e35657876cecfe606c5118c281f0d3c2a1573 WatchSource:0}: Error finding container 6ee709430472e7662db34382f38e35657876cecfe606c5118c281f0d3c2a1573: Status 404 returned error can't find the container with id 6ee709430472e7662db34382f38e35657876cecfe606c5118c281f0d3c2a1573
Nov 03 20:03:22.906766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.906733    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:22.906830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.906801    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:22.906863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:22.906830    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:23.114081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:23.113990    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:23.742621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:23.742312    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" event={"ID":"33a96be7-05f9-4135-be1d-9ab1bb17948a","Type":"ContainerStarted","Data":"fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005"}
Nov 03 20:03:23.742621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:23.742382    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" event={"ID":"33a96be7-05f9-4135-be1d-9ab1bb17948a","Type":"ContainerStarted","Data":"237f965007ff9f2097955ea8433a7c7841ce2965ec178ae386cf683ffddf6888"}
Nov 03 20:03:23.745132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:23.745098    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" event={"ID":"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc","Type":"ContainerStarted","Data":"3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e"}
Nov 03 20:03:23.745315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:23.745145    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" event={"ID":"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc","Type":"ContainerStarted","Data":"6ee709430472e7662db34382f38e35657876cecfe606c5118c281f0d3c2a1573"}
Nov 03 20:03:23.767034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:23.766994    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podStartSLOduration=1.766981317 podStartE2EDuration="1.766981317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:23.764594612 +0000 UTC m=+1396.433796082" watchObservedRunningTime="2026-11-03 20:03:23.766981317 +0000 UTC m=+1396.436182817"
Nov 03 20:03:23.800225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:23.792747    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podStartSLOduration=2.792725321 podStartE2EDuration="2.792725321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:23.789674527 +0000 UTC m=+1396.458875999" watchObservedRunningTime="2026-11-03 20:03:23.792725321 +0000 UTC m=+1396.461926793"
Nov 03 20:03:24.233372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.226098    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:24.570372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.570271    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:24.574382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.574343    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:03:24.574666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.574617    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:03:24.618776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.618698    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:24.642234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.636451    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:03:24.642234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.636512    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:03:24.961057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:24.957677    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:03:25.155227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.154374    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:25.563219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.562662    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:03:25.563219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.562725    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:03:25.616224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.615828    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:03:25.616224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.615901    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:03:25.620234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.616609    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:25.962377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.962276    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:25.962957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:25.962931    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:03:26.025222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.022190    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:26.203218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.200145    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:26.203218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.200899    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:03:26.203218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.202276    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:26.353221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.352998    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:26.372228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.369126    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:03:26.461009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.460974    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:03:26.562847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.562806    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:03:26.563016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.562881    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:03:26.623668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.623452    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:03:26.623668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:26.623530    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:03:27.511223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.510903    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:27.562863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.562820    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:03:27.563064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.562886    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:03:27.599220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.598645    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:27.617221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.611551    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:03:27.617221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.611614    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:03:27.643319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.643291    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:27.956559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:27.956478    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:28.498223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:28.497489    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:28.564001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:28.563964    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:03:28.564317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:28.564277    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:03:28.609802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:28.609751    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:03:28.610088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:28.610037    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:03:29.462816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:29.462778    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:29.562582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:29.562541    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:03:29.562842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:29.562815    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 03 20:03:29.609655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:29.609612    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:03:29.609925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:29.609894    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:03:30.308006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:30.307974    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:30.610370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:30.610271    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 20:03:30.610370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:30.610345    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 20:03:31.130003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.129957    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.347375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.347340    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:31.347805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.347778    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:31.349213 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.349175    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:03:31.451589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.451497    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:31.646233 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:03:31.645855    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:03:31.735756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.735724    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:31.736144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.736109    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:31.737041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:31.737020    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:03:32.090641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:32.090562    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:32.805425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:32.805388    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:33.558964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:33.558928    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:34.296689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:34.296656    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:35.130288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:35.123673    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:35.896390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:35.896357    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:36.827679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:36.827643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:37.090235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.090132    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"]
Nov 03 20:03:37.094247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.094227    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.105404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.105380    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"]
Nov 03 20:03:37.194371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194338    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194385    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194470    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2ee665-2396-4632-b5eb-8ee520151e07-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194519    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194635    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194666    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194760    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194792    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.194847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.194823    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6m4\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-kube-api-access-fx6m4\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296397    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296447    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296510    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296540    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296580    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6m4\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-kube-api-access-fx6m4\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296638    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296682    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296725    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2ee665-2396-4632-b5eb-8ee520151e07-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.296830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296770    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.297036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296882    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.297036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.296993    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.297148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.297088    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.297256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.297140    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.297478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.297459    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2ee665-2396-4632-b5eb-8ee520151e07-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.298879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.298857    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.299286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.299269    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.304496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.304471    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6m4\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-kube-api-access-fx6m4\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.304598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.304513    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.409631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.409539    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:37.621120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.621047    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:37.703042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.702962    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"]
Nov 03 20:03:37.709932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.709905    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"]
Nov 03 20:03:37.712411 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:37.712383    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2ee665_2396_4632_b5eb_8ee520151e07.slice/crio-5cfcd4e9b4919af77055f8322df0e31b98e0e772b8c72669a43253b6e39c8d43 WatchSource:0}: Error finding container 5cfcd4e9b4919af77055f8322df0e31b98e0e772b8c72669a43253b6e39c8d43: Status 404 returned error can't find the container with id 5cfcd4e9b4919af77055f8322df0e31b98e0e772b8c72669a43253b6e39c8d43
Nov 03 20:03:37.715296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.715258    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:37.715401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.715325    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:37.715401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.715358    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:37.715900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.715880    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.723515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.723485    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"openshift-service-ca.crt\""
Nov 03 20:03:37.723701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.723685    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"kube-root-ca.crt\""
Nov 03 20:03:37.724244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.724221    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"default-dockercfg-8tctz\""
Nov 03 20:03:37.724339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.724317    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"distinct-w1-b42aea-kserve-self-signed-certs\""
Nov 03 20:03:37.724381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.724333    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"]
Nov 03 20:03:37.789916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.789892    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:37.804291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.804257    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-dshm\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.804474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.804448    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7m8h\" (UniqueName: \"kubernetes.io/projected/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-kube-api-access-r7m8h\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.804540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.804505    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-home\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.804709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.804686    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-model-cache\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.804759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.804743    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tmp-dir\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.804866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.804829    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tls-certs\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.908074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.908009    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-model-cache\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.908959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.908923    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-model-cache\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.911062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.911016    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tmp-dir\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.912062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.912043    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tmp-dir\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.913793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.913775    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tls-certs\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.916471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.916423    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tls-certs\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.917535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.917514    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-dshm\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.921095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.920752    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-dshm\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.921684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.921663    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7m8h\" (UniqueName: \"kubernetes.io/projected/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-kube-api-access-r7m8h\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.922269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.922250    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-home\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.922760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.922647    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-home\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:37.950231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:37.948716    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7m8h\" (UniqueName: \"kubernetes.io/projected/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-kube-api-access-r7m8h\") pod \"distinct-w1-b42aea-kserve-b88b7f8fc-msb52\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:38.009566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.009529    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" event={"ID":"9f2ee665-2396-4632-b5eb-8ee520151e07","Type":"ContainerStarted","Data":"f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54"}
Nov 03 20:03:38.009746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.009590    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" event={"ID":"9f2ee665-2396-4632-b5eb-8ee520151e07","Type":"ContainerStarted","Data":"5cfcd4e9b4919af77055f8322df0e31b98e0e772b8c72669a43253b6e39c8d43"}
Nov 03 20:03:38.031467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.031408    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podStartSLOduration=1.031392672 podStartE2EDuration="1.031392672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:38.028224046 +0000 UTC m=+1410.697425516" watchObservedRunningTime="2026-11-03 20:03:38.031392672 +0000 UTC m=+1410.700594143"
Nov 03 20:03:38.050508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.044417    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:38.335041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.335005    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"]
Nov 03 20:03:38.337370 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:38.337339    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b82d548_716d_4a3e_98f1_0b3c8e4c91b0.slice/crio-56b2db2588d163d7a673f6da820d5cf94521924a98e6a67668ba2b93b1596a4d WatchSource:0}: Error finding container 56b2db2588d163d7a673f6da820d5cf94521924a98e6a67668ba2b93b1596a4d: Status 404 returned error can't find the container with id 56b2db2588d163d7a673f6da820d5cf94521924a98e6a67668ba2b93b1596a4d
Nov 03 20:03:38.409977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.409951    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:38.411399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.411375    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:38.411510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.411429    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 20:03:38.456227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.452747    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.858306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.858277    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"]
Nov 03 20:03:38.863918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.863893    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:38.869387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.869362    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ab1e46\"/\"distinct-w5-ab1e46-kserve-self-signed-certs\""
Nov 03 20:03:38.870909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.870888    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"]
Nov 03 20:03:38.936567 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.936537    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:38.941979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.941956    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tmp-dir\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:38.942079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.942008    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-model-cache\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:38.942153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.942136    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfkz\" (UniqueName: \"kubernetes.io/projected/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-kube-api-access-2lfkz\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:38.942189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.942178    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-dshm\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:38.942243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.942214    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tls-certs\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:38.942364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:38.942350    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-home\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.049222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.049024    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfkz\" (UniqueName: \"kubernetes.io/projected/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-kube-api-access-2lfkz\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.049222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.049101    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-dshm\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.049222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.049124    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tls-certs\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.050227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.049566    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-home\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.050227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.049620    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tmp-dir\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.050227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.049688    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-model-cache\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.050227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.050137    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-model-cache\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.050550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.050449    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-home\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.050853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.050702    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tmp-dir\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.067265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.063115    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tls-certs\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.077219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.068548    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfkz\" (UniqueName: \"kubernetes.io/projected/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-kube-api-access-2lfkz\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.118464 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.115611    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-dshm\") pod \"distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") " pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.127238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.125611    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" event={"ID":"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0","Type":"ContainerStarted","Data":"e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853"}
Nov 03 20:03:39.127238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.125750    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" event={"ID":"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0","Type":"ContainerStarted","Data":"56b2db2588d163d7a673f6da820d5cf94521924a98e6a67668ba2b93b1596a4d"}
Nov 03 20:03:39.149292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.149241    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podStartSLOduration=2.149225283 podStartE2EDuration="2.149225283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:39.148532358 +0000 UTC m=+1411.817733842" watchObservedRunningTime="2026-11-03 20:03:39.149225283 +0000 UTC m=+1411.818426747"
Nov 03 20:03:39.205092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.205049    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:39.331577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.330911    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:39.420293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.419640    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:39.420293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.419698    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 20:03:39.501342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:39.501306    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"]
Nov 03 20:03:39.503808 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:39.503783    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d36fc54_8a2c_45c0_9b1a_b7d441c1d947.slice/crio-6966f5fb6e814c12fcaff10b1a9d9e433f2cc93f6e22e02e1807baa5efe1bce5 WatchSource:0}: Error finding container 6966f5fb6e814c12fcaff10b1a9d9e433f2cc93f6e22e02e1807baa5efe1bce5: Status 404 returned error can't find the container with id 6966f5fb6e814c12fcaff10b1a9d9e433f2cc93f6e22e02e1807baa5efe1bce5
Nov 03 20:03:40.046091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.046057    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:40.139811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.139769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:40.140288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.140257    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" event={"ID":"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947","Type":"ContainerStarted","Data":"042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326"}
Nov 03 20:03:40.140438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.140307    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" event={"ID":"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947","Type":"ContainerStarted","Data":"6966f5fb6e814c12fcaff10b1a9d9e433f2cc93f6e22e02e1807baa5efe1bce5"}
Nov 03 20:03:40.177608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.177536    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podStartSLOduration=2.177516506 podStartE2EDuration="2.177516506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:40.174601451 +0000 UTC m=+1412.843802921" watchObservedRunningTime="2026-11-03 20:03:40.177516506 +0000 UTC m=+1412.846717977"
Nov 03 20:03:40.208987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.208942    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:40.410700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.410609    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:40.411846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.411809    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 20:03:40.561240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:40.561206    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:03:41.001886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:41.001846    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:41.410585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:41.410499    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:41.410585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:41.410554    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 20:03:42.417220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:42.415599    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:42.417220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:42.415674    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 20:03:42.440701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:42.440577    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:42.986342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:42.957485    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:42.986342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:42.958949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:43.289390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:43.289286    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 03 20:03:43.415223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:43.414587    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:43.415223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:43.414651    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 20:03:43.463324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:43.463278    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:44.102948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:44.098733    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:44.411004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:44.410905    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 20:03:44.411004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:44.410962    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 20:03:44.978886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:44.978855    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.760260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:45.760223    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:45.873376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:45.873340    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:46.215798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:46.215702    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:46.216415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:46.216388    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:46.217115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:46.217094    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:03:46.732735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:46.731318    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:47.413369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:47.413339    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.046482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:48.046450    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:48.059319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:48.059293    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:48.098712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:48.098682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:48.257144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:48.257114    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:48.795988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:48.795959    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:49.206137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:49.206051    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:49.218544 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:49.218523    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:49.264992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:49.264966    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:49.596365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:49.596325    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:50.387169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:50.387130    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:51.096487 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:51.096449    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:51.883926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:51.883892    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:52.236019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.235981    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"]
Nov 03 20:03:52.245151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.245124    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.254410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.254380    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"]
Nov 03 20:03:52.359422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359384    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359428    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359544    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364a31d7-5518-433e-83ca-5d873b133955-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359565    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359592    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlgj7\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-kube-api-access-hlgj7\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359673    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359742    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364a31d7-5518-433e-83ca-5d873b133955-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359777    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.359879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.359864    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.455398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.455364    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"]
Nov 03 20:03:52.459737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.459715    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.460606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460578    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460679    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364a31d7-5518-433e-83ca-5d873b133955-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460716    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460757    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgj7\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-kube-api-access-hlgj7\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460823    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460876    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364a31d7-5518-433e-83ca-5d873b133955-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460904    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.460958    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.461035    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.461339    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.461391    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.462181    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364a31d7-5518-433e-83ca-5d873b133955-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.462364    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.462779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.462603    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.464413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.464390    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.481313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.481284    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364a31d7-5518-433e-83ca-5d873b133955-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.483721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.483694    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgj7\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-kube-api-access-hlgj7\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.484249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.484217    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"]
Nov 03 20:03:52.484529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.484508    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.562137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562109    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562162    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8e93e9-5642-420b-96df-068346841dd4-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562320    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-kube-api-access-cs7n7\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562389    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562413    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562448    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562547    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562604    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8e93e9-5642-420b-96df-068346841dd4-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.562685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.562672    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.567998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.567976    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:52.606748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.606672    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.689554    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.689629    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.689691    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.689736    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.689774    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8e93e9-5642-420b-96df-068346841dd4-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.689953    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.690020    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.690088    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8e93e9-5642-420b-96df-068346841dd4-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.690129    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-kube-api-access-cs7n7\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.696224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.690989    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.701286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.701057    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-kube-api-access-cs7n7\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.715225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.701876    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.715225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.702219    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.715225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.702454    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.715225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.703051    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8e93e9-5642-420b-96df-068346841dd4-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.728238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.727235    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.728238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.727632    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8e93e9-5642-420b-96df-068346841dd4-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.729091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.729070    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.818984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.818908    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:52.848332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.848305    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"]
Nov 03 20:03:52.850262 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:52.850234    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364a31d7_5518_433e_83ca_5d873b133955.slice/crio-1edfe4ece6c8b914b9f44ba662970526827289b40c2c32f945b76bffeed4bdc1 WatchSource:0}: Error finding container 1edfe4ece6c8b914b9f44ba662970526827289b40c2c32f945b76bffeed4bdc1: Status 404 returned error can't find the container with id 1edfe4ece6c8b914b9f44ba662970526827289b40c2c32f945b76bffeed4bdc1
Nov 03 20:03:52.852949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.852914    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:52.853021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.852983    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:52.853021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:52.853013    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:53.104120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.104093    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"]
Nov 03 20:03:53.104860 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:03:53.104835    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8e93e9_5642_420b_96df_068346841dd4.slice/crio-20b7b76ddd5607d1b8891a405958eb4fbdb30c8e3c70a32039e9f2fc1aaa6684 WatchSource:0}: Error finding container 20b7b76ddd5607d1b8891a405958eb4fbdb30c8e3c70a32039e9f2fc1aaa6684: Status 404 returned error can't find the container with id 20b7b76ddd5607d1b8891a405958eb4fbdb30c8e3c70a32039e9f2fc1aaa6684
Nov 03 20:03:53.107262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.107232    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:53.107330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.107295    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:53.107330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.107322    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:03:53.285530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.285260    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" event={"ID":"ff8e93e9-5642-420b-96df-068346841dd4","Type":"ContainerStarted","Data":"947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2"}
Nov 03 20:03:53.285530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.285359    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" event={"ID":"ff8e93e9-5642-420b-96df-068346841dd4","Type":"ContainerStarted","Data":"20b7b76ddd5607d1b8891a405958eb4fbdb30c8e3c70a32039e9f2fc1aaa6684"}
Nov 03 20:03:53.287336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.287311    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" event={"ID":"364a31d7-5518-433e-83ca-5d873b133955","Type":"ContainerStarted","Data":"9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a"}
Nov 03 20:03:53.287457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.287348    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" event={"ID":"364a31d7-5518-433e-83ca-5d873b133955","Type":"ContainerStarted","Data":"1edfe4ece6c8b914b9f44ba662970526827289b40c2c32f945b76bffeed4bdc1"}
Nov 03 20:03:53.306798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.306750    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podStartSLOduration=1.306737286 podStartE2EDuration="1.306737286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:53.30471344 +0000 UTC m=+1425.973914911" watchObservedRunningTime="2026-11-03 20:03:53.306737286 +0000 UTC m=+1425.975938756"
Nov 03 20:03:53.340224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.320918    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:53.340224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.331999    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podStartSLOduration=1.331979204 podStartE2EDuration="1.331979204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:53.326172301 +0000 UTC m=+1425.995373767" watchObservedRunningTime="2026-11-03 20:03:53.331979204 +0000 UTC m=+1426.001180674"
Nov 03 20:03:53.568582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.568545    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:03:53.569856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.569823    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:53.569963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.569898    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:03:53.661735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.661695    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:53.822428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:53.822325    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:03:54.341716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:54.341679    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:54.568522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:54.568487    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:54.568693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:54.568542    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:03:54.825221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:54.824309    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:54.825221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:54.824374    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 20:03:55.569225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:55.568669    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:55.569225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:55.568732    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:03:55.815222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:55.814738    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:55.838283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:55.824555    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:55.838283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:55.824624    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 20:03:55.927223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:55.927008    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:55.955222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:55.954315    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:56.179282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.178288    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:03:56.257223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.254335    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:56.257223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.256327    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:03:56.375221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.374381    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:56.462284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.461289    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:56.574219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.573650    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:56.574219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.573710    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:03:56.827339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.827297    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:56.827630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.827599    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 20:03:56.960248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.959291    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:03:56.960248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:56.959996    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:57.046256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.045310    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:57.333387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.232429    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:57.333387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.233047    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:03:57.333387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.271287    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:57.333387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.271671    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:03:57.333387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.272162    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:57.333387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.272303    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:57.549232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.545041    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:57.593399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.556882    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:03:57.982730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:57.982269    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:03:58.066221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.064582    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:03:58.146665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.144676    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:03:58.147156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.147130    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:03:58.167055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.157934    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:58.167324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.167297    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 20:03:58.232627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.232585    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:58.249923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.249842    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:58.254069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.254028    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:03:58.254278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.253981    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:03:58.254428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.254416    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:03:58.254521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.254512    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:03:58.254618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.254609    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:03:58.254739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.254727    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:03:58.260105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.260083    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:03:58.263380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.263362    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:03:58.268562 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.268538    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:03:58.341479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.341423    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:58.344945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.344843    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:03:58.568346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.568264    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:58.568346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.568329    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:03:58.823271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.822586    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:58.823271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:58.822664    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 20:03:59.360972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:59.360938    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:03:59.569130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:59.569097    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:03:59.569293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:59.569151    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:03:59.824229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:59.820606    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:03:59.824229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:03:59.820668    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 20:04:00.193150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:00.193050    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:00.569039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:00.569007    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 20:04:00.569220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:00.569063    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 20:04:00.820362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:00.820274    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 20:04:00.820362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:00.820334    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 20:04:01.167904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:01.167817    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:01.619374 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:04:01.619337    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:04:01.657918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:01.657885    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:04:01.659322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:01.659283    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:04:01.659847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:01.659827    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:04:02.089159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.089124    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:02.511547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.511501    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"]
Nov 03 20:04:02.517239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.517176    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.527644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.525519    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"]
Nov 03 20:04:02.527644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.525639    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b87733\"/\"distinct-2-w3-b87733-kserve-self-signed-certs\""
Nov 03 20:04:02.541370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.540525    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-dshm\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.541370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.540635    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-home\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.541370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.540705    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-model-cache\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.541370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.540772    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tmp-dir\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.541370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.540925    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tls-certs\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.541370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.540959    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvh5\" (UniqueName: \"kubernetes.io/projected/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-kube-api-access-2fvh5\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.587503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.587464    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:02.642071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642039    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-home\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642093    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-model-cache\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642134    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tmp-dir\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642251    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tls-certs\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642267    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvh5\" (UniqueName: \"kubernetes.io/projected/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-kube-api-access-2fvh5\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642290    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-dshm\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642846    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-home\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642873    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tmp-dir\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.642978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.642920    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-model-cache\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.644669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.644644    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-dshm\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.644821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.644803    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tls-certs\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.707243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.702128    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvh5\" (UniqueName: \"kubernetes.io/projected/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-kube-api-access-2fvh5\") pod \"distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") " pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:02.795270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.793580    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:04:02.795270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.794540    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:04:02.798836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.798799    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:04:02.843681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:02.842230    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:03.006116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:03.006083    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:03.138390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:03.138309    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"]
Nov 03 20:04:03.143218 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:03.143171    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d1a2ac4_29ea_48fe_b139_d33c7b4a261b.slice/crio-6af24c78cc83afdd53ef94e43a84511d879bdf0f46da00f61bf3dbbe0d2bd9d8 WatchSource:0}: Error finding container 6af24c78cc83afdd53ef94e43a84511d879bdf0f46da00f61bf3dbbe0d2bd9d8: Status 404 returned error can't find the container with id 6af24c78cc83afdd53ef94e43a84511d879bdf0f46da00f61bf3dbbe0d2bd9d8
Nov 03 20:04:03.190248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:03.190175    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" event={"ID":"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b","Type":"ContainerStarted","Data":"6af24c78cc83afdd53ef94e43a84511d879bdf0f46da00f61bf3dbbe0d2bd9d8"}
Nov 03 20:04:03.695994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:03.695960    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.197800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:04.197763    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" event={"ID":"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b","Type":"ContainerStarted","Data":"5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5"}
Nov 03 20:04:04.685099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:04.685065    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:04.842326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:04.842286    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:05.390572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:05.390536    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:06.112528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:06.112492    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:06.778800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:06.778765    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:06.953134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:06.953102    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:07.083269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.073294    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podStartSLOduration=5.073273953 podStartE2EDuration="5.073273953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:04.2405511 +0000 UTC m=+1436.909752562" watchObservedRunningTime="2026-11-03 20:04:07.073273953 +0000 UTC m=+1439.742475423"
Nov 03 20:04:07.083269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.073523    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"]
Nov 03 20:04:07.178515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.178007    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.214836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.214804    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"]
Nov 03 20:04:07.218582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.218554    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"]
Nov 03 20:04:07.222664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.222644    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"]
Nov 03 20:04:07.222779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.222743    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.253660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.253629    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.253798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.253666    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.253798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.253690    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.253798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.253748    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skn47\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-kube-api-access-skn47\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.253906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.253872    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.253939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.253903    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.253939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.253926    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.254019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254002    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nvt\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-kube-api-access-92nvt\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.254076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254065    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.254110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254085    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.254110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254100    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.254171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254119    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.254230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254215    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.254269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254247    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c977e2-140f-4591-9f8c-78361d36ef0f-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.254331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254318    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.254374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254363    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.254407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254386    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.254442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.254409    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.314035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.314001    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"]
Nov 03 20:04:07.318129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.318110    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.331613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.331587    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"]
Nov 03 20:04:07.356307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356221    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.356307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356270    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356309    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356336    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356377    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356400    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b10504a-edeb-4638-86aa-52e54efea88b-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356423    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b10504a-edeb-4638-86aa-52e54efea88b-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356452    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nvt\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-kube-api-access-92nvt\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356517    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356546    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356571    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356600    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.356660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356641    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356665    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356711    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c977e2-140f-4591-9f8c-78361d36ef0f-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356735    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356794    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356823    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356892    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356928    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.356961    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.357001    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.357046    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbv7j\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-kube-api-access-gbv7j\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.357127    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.357150    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.357175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.357174    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.357612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.357227    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skn47\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-kube-api-access-skn47\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.357958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.357935    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.358180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.358163    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.358493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.358475    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.358671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.358650    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"]
Nov 03 20:04:07.359632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.359601    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.360929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.360902    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c977e2-140f-4591-9f8c-78361d36ef0f-istiod-ca-cert\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.361380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.361353    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-certs\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.361473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.361441    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.361652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.361623    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-credential-socket\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.361926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.361695    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.361926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.361784    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.361926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.361882    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-data\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.362039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.361962    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-envoy\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.362715 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.362698    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.363573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.363557    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-podinfo\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.364906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.364886    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.377346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.377319    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nvt\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-kube-api-access-92nvt\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.379269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.378300    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"]
Nov 03 20:04:07.380487 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.380467    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-token\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.382324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.382303    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.382666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.382646    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skn47\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-kube-api-access-skn47\") pod \"e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") " pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.458599 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458569    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.458743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458604    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.458743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458636    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbv7j\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-kube-api-access-gbv7j\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.458743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458678    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.458743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458700    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.458882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458778    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.458882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458817    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.458882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458835    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.458981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458882    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.458981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458911    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b10504a-edeb-4638-86aa-52e54efea88b-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.458981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.458945    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b10504a-edeb-4638-86aa-52e54efea88b-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459031    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459090    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzlkj\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-kube-api-access-wzlkj\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.459256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459120    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459142    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.459256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459179    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459287    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459321    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.459546 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459441    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459546 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459533    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459647    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b10504a-edeb-4638-86aa-52e54efea88b-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459700    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.459783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.459758    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.461100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.461080    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.461436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.461420    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b10504a-edeb-4638-86aa-52e54efea88b-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.467117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.467089    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.467228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.467189    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbv7j\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-kube-api-access-gbv7j\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.512584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.512555    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:07.543265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.543239    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:07.561295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561260    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzlkj\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-kube-api-access-wzlkj\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561314    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561403    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561544 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561457    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561532    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561577    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561606    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561648    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561731    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561798    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-data\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.561865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.561847    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.562056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.562032    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-certs\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.562325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.562304    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-credential-socket\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.562497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.562478    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istiod-ca-cert\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.564147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.564127    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-envoy\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.564580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.564555    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-podinfo\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.584376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.569966    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-token\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.584376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.576787    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzlkj\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-kube-api-access-wzlkj\") pod \"e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") " pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.633615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.633586    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:07.707626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.707589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:07.717701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.717487    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:07.820272 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:07.819713    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3416cbfd_53c7_494a_81e5_ec52c63d39d6.slice/crio-160acb3db6caa4f427da0f82a2f71b36bd81b8ade037ba5cd89efdca2f3d7929 WatchSource:0}: Error finding container 160acb3db6caa4f427da0f82a2f71b36bd81b8ade037ba5cd89efdca2f3d7929: Status 404 returned error can't find the container with id 160acb3db6caa4f427da0f82a2f71b36bd81b8ade037ba5cd89efdca2f3d7929
Nov 03 20:04:07.820446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.820421    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"]
Nov 03 20:04:07.827799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.826994    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.827799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.827083    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.827799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.827130    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.842756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.842621    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"]
Nov 03 20:04:07.853455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.852684    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.853455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.852775    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.853455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.852818    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.928528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.928395    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"]
Nov 03 20:04:07.931829 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:07.931797    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b10504a_edeb_4638_86aa_52e54efea88b.slice/crio-e6ae26a5efc324f2f6996376a98ce92da5b667d1c00766b6dad3349e8516df43 WatchSource:0}: Error finding container e6ae26a5efc324f2f6996376a98ce92da5b667d1c00766b6dad3349e8516df43: Status 404 returned error can't find the container with id e6ae26a5efc324f2f6996376a98ce92da5b667d1c00766b6dad3349e8516df43
Nov 03 20:04:07.935519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.935475    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.935622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.935565    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:07.935622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:07.935610    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:08.109793 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:08.109220    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc9e6e9_6e8e_42c9_bb88_7d4f7fd8a2fa.slice/crio-879741170f4348c1108f234e522d86504f675bb80bdb3c41621b510b143eaf13 WatchSource:0}: Error finding container 879741170f4348c1108f234e522d86504f675bb80bdb3c41621b510b143eaf13: Status 404 returned error can't find the container with id 879741170f4348c1108f234e522d86504f675bb80bdb3c41621b510b143eaf13
Nov 03 20:04:08.121525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.117043    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"]
Nov 03 20:04:08.132143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.132099    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:08.132315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.132190    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:08.132394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.132347    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:08.255161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.254060    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" event={"ID":"3416cbfd-53c7-494a-81e5-ec52c63d39d6","Type":"ContainerStarted","Data":"beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a"}
Nov 03 20:04:08.255161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.254159    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" event={"ID":"3416cbfd-53c7-494a-81e5-ec52c63d39d6","Type":"ContainerStarted","Data":"160acb3db6caa4f427da0f82a2f71b36bd81b8ade037ba5cd89efdca2f3d7929"}
Nov 03 20:04:08.265418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.265383    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" event={"ID":"0b10504a-edeb-4638-86aa-52e54efea88b","Type":"ContainerStarted","Data":"30fc5d65e6909017c91360c2b41069b6b07ed343dde60ecb5c8946badcbc126d"}
Nov 03 20:04:08.265657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.265446    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" event={"ID":"0b10504a-edeb-4638-86aa-52e54efea88b","Type":"ContainerStarted","Data":"e6ae26a5efc324f2f6996376a98ce92da5b667d1c00766b6dad3349e8516df43"}
Nov 03 20:04:08.267500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.267474    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" event={"ID":"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa","Type":"ContainerStarted","Data":"f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65"}
Nov 03 20:04:08.267611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.267524    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" event={"ID":"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa","Type":"ContainerStarted","Data":"879741170f4348c1108f234e522d86504f675bb80bdb3c41621b510b143eaf13"}
Nov 03 20:04:08.270803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.270781    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" event={"ID":"32c977e2-140f-4591-9f8c-78361d36ef0f","Type":"ContainerStarted","Data":"c7f738898274e30450c09fc31dab84a61f4d7049166afc8f4beeca2b0951fea4"}
Nov 03 20:04:08.270932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.270825    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" event={"ID":"32c977e2-140f-4591-9f8c-78361d36ef0f","Type":"ContainerStarted","Data":"1814a32c1750bea109628c2ba48dfa3c4b487d082d738058db83974efe651035"}
Nov 03 20:04:08.295220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.280500    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podStartSLOduration=1.280482721 podStartE2EDuration="1.280482721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.277681024 +0000 UTC m=+1440.946882497" watchObservedRunningTime="2026-11-03 20:04:08.280482721 +0000 UTC m=+1440.949684191"
Nov 03 20:04:08.340820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.339917    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podStartSLOduration=1.339896724 podStartE2EDuration="1.339896724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.317685348 +0000 UTC m=+1440.986886824" watchObservedRunningTime="2026-11-03 20:04:08.339896724 +0000 UTC m=+1441.009098196"
Nov 03 20:04:08.341912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.341872    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podStartSLOduration=1.341861572 podStartE2EDuration="1.341861572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.339389033 +0000 UTC m=+1441.008590504" watchObservedRunningTime="2026-11-03 20:04:08.341861572 +0000 UTC m=+1441.011063043"
Nov 03 20:04:08.522221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.521542    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:08.527112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.523638    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:08.527112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.523692    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:08.552229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.547211    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:08.552229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.548477    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:08.552229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.548531    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:08.625233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.622684    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:08.646221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.634304    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:08.721335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:08.721301    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:09.521222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.520837    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.521222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.520896    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:09.554222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.546634    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.554222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.546698    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:09.659223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.659137    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.659223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.659213    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:09.711216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.702219    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:09.732064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.732029    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:09.732351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:09.732317    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:10.232227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.216556    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:04:10.232227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.217717    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 03 20:04:10.232711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.232668    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:10.263336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.261819    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:10.266262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.265501    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:10.287221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.274289    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:10.360342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.360297    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:10.421221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.393878    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:04:10.518388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.518299    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.518656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.518627    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:10.576226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.576093    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.576226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.576150    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:10.639236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.636581    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.639236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.636644    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:10.723222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.722351    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:10.723222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.722413    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:10.908613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.908523    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:10.924338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.924290    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:10.996414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:10.996367    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:11.219404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.219296    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:11.220446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.220417    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:04:11.264225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.259280    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:04:11.264225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.260177    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:04:11.268222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.266318    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:11.268222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.266649    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:04:11.277298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.277263    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:11.277797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.277777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:04:11.296223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.293941    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podStartSLOduration=4.293920533 podStartE2EDuration="4.293920533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:08.36930855 +0000 UTC m=+1441.038510014" watchObservedRunningTime="2026-11-03 20:04:11.293920533 +0000 UTC m=+1443.963122003"
Nov 03 20:04:11.470296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.455320    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:11.487218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.482904    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:04:11.499370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.498723    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:11.560226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.533888    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.560226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.533967    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:11.676131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.674481    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.676131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.674546    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:11.792233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.791720    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.792233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.791810    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:11.877636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.877599    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:11.877957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.877912    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:11.887223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.883246    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:04:11.973577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.973520    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:11.978365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:11.978329    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:04:12.232288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.226865    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:12.301224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.298176    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:04:12.327226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.326841    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:12.451229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.418012    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:04:12.451229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.440023    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:04:12.487643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.481595    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:04:12.570223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.526056    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:04:12.570223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.529735    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:12.570223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.542441    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.570223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.542569    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:12.714243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.676238    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:12.955274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.696097    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.955528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.955492    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:12.955657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.929226    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:12.955919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.696230    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.956272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.956248    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:12.956427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.755374    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:12.956581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.956564    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:12.987021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:12.986811    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:13.231222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.227276    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:13.318862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.314434    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:04:13.400228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.399443    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 03 20:04:13.437223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.432310    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:13.461223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.444324    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:13.488395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.487587    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:04:13.492837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.492756    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:13.519221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.511960    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:04:13.519221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.518890    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:13.519458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.519270    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.519458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.519314    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:13.538221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.531295    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:13.538221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.531731    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:13.559222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.554283    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.559222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.554339    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:13.639224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.638340    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.639224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.638410    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:13.724684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.724626    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.725271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.725234    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:13.939979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:13.939877    5282 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Nov 03 20:04:14.023109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.020371    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:04:14.251285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.230287    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:14.327224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.322299    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:14.443223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.442689    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:14.449227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.448373    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:04:14.501222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.493999    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:04:14.513655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.513561    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.513898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.513869    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:14.540229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.539288    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:14.540229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.539687    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:04:14.562222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.556243    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.562222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.556297    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:14.606224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.603877    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:04:14.665223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.641790    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:14.665223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.642242    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.665223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.642293    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:14.720230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.718758    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.756237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:14.720486    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:15.032833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.032782    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:15.263239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.236301    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:04:15.340424 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.340304    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:04:15.355404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.355300    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:04:15.472254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.466279    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:15.472254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.469707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:15.506469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.504555    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:15.561652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.518130    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.561652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.525496    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:15.561652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.541163    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:15.589222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.562171    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.589222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.562249    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 20:04:15.707213 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.659957    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:15.833622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.833586    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:04:15.839346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.839313    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:04:15.839925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.839879    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.840034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.839951    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:15.856430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.856384    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.856620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.856463    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:15.868569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.868534    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:04:15.913339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.913308    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:04:15.960547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.960467    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:04:15.985228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:15.985137    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:04:16.348304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.348268    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:16.521954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.521910    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.522223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.522174    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 20:04:16.547222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.536462    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"]
Nov 03 20:04:16.566151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.566116    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"]
Nov 03 20:04:16.576480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.566326    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.586502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.586476    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"kube-root-ca.crt\""
Nov 03 20:04:16.586791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.586774    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"distinct-2-w2-11a5b0-kserve-self-signed-certs\""
Nov 03 20:04:16.587013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.586998    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"default-dockercfg-rm8q5\""
Nov 03 20:04:16.587189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.587174    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-11a5b0\"/\"openshift-service-ca.crt\""
Nov 03 20:04:16.594553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.594530    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:16.634767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.634692    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.634986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.634956    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 03 20:04:16.693269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.693228    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-home\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.693827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.693589    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-tmp-dir\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.694052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.694032    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885dd171-716d-4d0f-a798-2072f63c243c-tls-certs\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.694263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.694241    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-dshm\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.694603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.694572    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/885dd171-716d-4d0f-a798-2072f63c243c-kube-api-access-hg92x\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.694838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.694817    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-model-cache\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.750374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.750331    5282 patch_prober.go:29] interesting pod/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.750706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.750663    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 20:04:16.808817 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.808776    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/885dd171-716d-4d0f-a798-2072f63c243c-kube-api-access-hg92x\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.809190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.809145    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-model-cache\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.809466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.809444    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-home\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.809674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.809634    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-tmp-dir\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.809861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.809833    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885dd171-716d-4d0f-a798-2072f63c243c-tls-certs\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.810062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.810045    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-dshm\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.819600 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.818574    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-model-cache\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.819600 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.819170    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-home\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.846520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.843171    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-tmp-dir\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.881220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.877219    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/885dd171-716d-4d0f-a798-2072f63c243c-kube-api-access-hg92x\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:16.932029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:16.895915    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885dd171-716d-4d0f-a798-2072f63c243c-tls-certs\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:17.029669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.009136    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-dshm\") pod \"distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") " pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:17.157221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.156917    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:17.184364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.182614    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:17.184899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.184877    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:17.194829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.194804    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:04:17.237857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.234491    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:17.541864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.536968    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:17.557950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.557741    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:17.578490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.577742    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:04:17.652495 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.650121    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:17.653802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.653774    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:17.656243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.655265    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:04:17.733503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.733476    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:17.734109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.734078    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:17.735020 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.734998    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:04:17.754053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.754030    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"]
Nov 03 20:04:17.755123 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:17.755096    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod885dd171_716d_4d0f_a798_2072f63c243c.slice/crio-3292e8fe6958c6ec30b45b994d0216760a04112ae85818a6d1eda4771d69ee1b WatchSource:0}: Error finding container 3292e8fe6958c6ec30b45b994d0216760a04112ae85818a6d1eda4771d69ee1b: Status 404 returned error can't find the container with id 3292e8fe6958c6ec30b45b994d0216760a04112ae85818a6d1eda4771d69ee1b
Nov 03 20:04:17.954840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.954806    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" event={"ID":"885dd171-716d-4d0f-a798-2072f63c243c","Type":"ContainerStarted","Data":"b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d"}
Nov 03 20:04:17.955259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.954852    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" event={"ID":"885dd171-716d-4d0f-a798-2072f63c243c","Type":"ContainerStarted","Data":"3292e8fe6958c6ec30b45b994d0216760a04112ae85818a6d1eda4771d69ee1b"}
Nov 03 20:04:17.975589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:17.975540    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podStartSLOduration=1.975524664 podStartE2EDuration="1.975524664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:17.973899965 +0000 UTC m=+1450.643101434" watchObservedRunningTime="2026-11-03 20:04:17.975524664 +0000 UTC m=+1450.644726134"
Nov 03 20:04:18.005487 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:18.005457    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:18.237344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:18.237308    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:18.405682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:18.405645    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:18.817851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:18.817819    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:04:18.889084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:18.889050    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:19.887236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:19.879328    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:20.171105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:20.170979    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:21.730221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:21.729655    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:22.436110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.436063    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:04:22.459000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.458966    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"]
Nov 03 20:04:22.467815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.467775    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.474665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.474633    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"]
Nov 03 20:04:22.596250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.569416    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.596577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.596539    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvbf\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-kube-api-access-wwvbf\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.600879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.600849    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.601166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.601149    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.601394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.601362    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.601631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.601597    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed66650-0337-4a97-9540-4c584d1ddab9-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.601924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.601896    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.602076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.602058    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.602271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.602254    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.703894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.703802    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.703894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.703872    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.703935    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed66650-0337-4a97-9540-4c584d1ddab9-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704053    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704084    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704156    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704281    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704285    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704336    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704393    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvbf\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-kube-api-access-wwvbf\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704439    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704441    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704611    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.704766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.704736    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed66650-0337-4a97-9540-4c584d1ddab9-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.706848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.706826    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.706959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.706869    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.713728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.713703    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.713821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.713711    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvbf\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-kube-api-access-wwvbf\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.773757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.773700    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:22.819605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.819571    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:22.881219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.876710    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:22.913781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:22.913695    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:23.195545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:23.195520    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"]
Nov 03 20:04:23.195790 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:23.195765    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed66650_0337_4a97_9540_4c584d1ddab9.slice/crio-7fda505ea4d91412f40734491ce1149beabbc5fc5c953b6255e6fc1d28e771e7 WatchSource:0}: Error finding container 7fda505ea4d91412f40734491ce1149beabbc5fc5c953b6255e6fc1d28e771e7: Status 404 returned error can't find the container with id 7fda505ea4d91412f40734491ce1149beabbc5fc5c953b6255e6fc1d28e771e7
Nov 03 20:04:23.198251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:23.198219    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:23.198321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:23.198283    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:23.198321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:23.198311    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:23.426682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:23.426646    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:24.072229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:24.072151    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" event={"ID":"4ed66650-0337-4a97-9540-4c584d1ddab9","Type":"ContainerStarted","Data":"06f13aae047020c2fe68f8edbdda4e09e8e4378fdfbc36544cdd440c417ecfc1"}
Nov 03 20:04:24.072617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:24.072256    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" event={"ID":"4ed66650-0337-4a97-9540-4c584d1ddab9","Type":"ContainerStarted","Data":"7fda505ea4d91412f40734491ce1149beabbc5fc5c953b6255e6fc1d28e771e7"}
Nov 03 20:04:24.096024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:24.095976    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podStartSLOduration=2.0959639230000002 podStartE2EDuration="2.095963923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:24.092260664 +0000 UTC m=+1456.761462133" watchObservedRunningTime="2026-11-03 20:04:24.095963923 +0000 UTC m=+1456.765165392"
Nov 03 20:04:24.227896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:24.227852    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:24.822765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:24.822724    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:24.824432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:24.824410    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:24.824494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:24.824460    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:04:25.001369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:25.001325    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:25.697224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:25.696673    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:25.825222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:25.824884    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:25.825222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:25.824951    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:04:26.404756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:26.404720    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:26.823191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:26.823156    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:26.823569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:26.823225    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:04:27.104970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:27.104889    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:27.236294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:27.236257    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:27.248737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:27.248710    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:27.799134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:27.799076    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:27.834665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:27.834628    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:27.835231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:27.835182    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:04:28.172635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:28.172558    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:28.598790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:28.598704    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:28.830673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:28.830635    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:28.830972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:28.830942    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:04:29.400630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:29.400592    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:29.823309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:29.823274    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:04:29.823494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:29.823324    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Nov 03 20:04:30.380697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:30.380663    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:30.827247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:30.827218    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:30.827685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:30.827546    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:30.828254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:30.828235    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:04:31.574062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.574030    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:31.615016 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:04:31.614985    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:04:31.656157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.656128    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"]
Nov 03 20:04:31.660392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.660369    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.668817 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.668772    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5348ca\"/\"distinct-2-w4-5348ca-kserve-self-signed-certs\""
Nov 03 20:04:31.670057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.669999    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"]
Nov 03 20:04:31.698718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.698687    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-home\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.698823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.698731    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tmp-dir\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.698861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.698832    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-dshm\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.698892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.698861    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tls-certs\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.698950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.698933    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-model-cache\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.699044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.699030    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wv62\" (UniqueName: \"kubernetes.io/projected/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-kube-api-access-9wv62\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.728425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.728396    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:31.809222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.809056    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-dshm\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.809222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.809119    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tls-certs\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.809222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.809219    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-model-cache\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.809522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.809336    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wv62\" (UniqueName: \"kubernetes.io/projected/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-kube-api-access-9wv62\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.809577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.809516    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-home\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.809630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.809580    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tmp-dir\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.813218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.810110    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tmp-dir\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.820666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.820618    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-model-cache\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.846993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.828542    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-home\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.847616 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.834512    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-dshm\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.861221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.851287    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tls-certs\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.881964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.881930    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wv62\" (UniqueName: \"kubernetes.io/projected/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-kube-api-access-9wv62\") pod \"distinct-2-w4-5348ca-kserve-6ddf876458-svl5w\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") " pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:31.979995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:31.979958    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:32.098232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:32.098123    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:32.280375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:32.280345    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"]
Nov 03 20:04:32.285296 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:32.285269    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c41fa90_ab28_419d_a5a7_6e19f3e1ac22.slice/crio-d81f2b63a1ab57c329c9b46b2d4b1bdc23b57d861a798639b4595f14a13d2972 WatchSource:0}: Error finding container d81f2b63a1ab57c329c9b46b2d4b1bdc23b57d861a798639b4595f14a13d2972: Status 404 returned error can't find the container with id d81f2b63a1ab57c329c9b46b2d4b1bdc23b57d861a798639b4595f14a13d2972
Nov 03 20:04:32.815822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:32.815785    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:33.212535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:33.212447    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" event={"ID":"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22","Type":"ContainerStarted","Data":"c0dd5d7c0531938c118513b2ddf3bd9c9ec1d233d3d7b97d680653613a400ece"}
Nov 03 20:04:33.212535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:33.212499    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" event={"ID":"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22","Type":"ContainerStarted","Data":"d81f2b63a1ab57c329c9b46b2d4b1bdc23b57d861a798639b4595f14a13d2972"}
Nov 03 20:04:33.232685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:33.232637    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podStartSLOduration=2.232620352 podStartE2EDuration="2.232620352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:33.229760018 +0000 UTC m=+1465.898961489" watchObservedRunningTime="2026-11-03 20:04:33.232620352 +0000 UTC m=+1465.901821821"
Nov 03 20:04:33.699374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:33.699330    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:33.980129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:33.980103    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:34.437271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:34.436120    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:35.090491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:35.090457    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:35.889881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:35.889850    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:36.013178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:36.013139    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:37.006643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.006608    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:37.326831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.326747    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"]
Nov 03 20:04:37.331334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.331308    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.344858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.344800    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"]
Nov 03 20:04:37.397977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.397938    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.397989    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.398048    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.398076    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.398211    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.398274    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.398292    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e98860-af8e-49cb-be50-efe9bb8a8a27-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.398392    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.398524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.398460    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kld42\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-kube-api-access-kld42\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.491289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.491244    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"]
Nov 03 20:04:37.495559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.495518    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.499784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.499752    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.499925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.499873    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.499925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.499901    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e98860-af8e-49cb-be50-efe9bb8a8a27-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.500045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.499988    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.500099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.500086    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kld42\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-kube-api-access-kld42\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.500180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.500134    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.500331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.500192    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.500331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.500247    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.500331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.500295    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.510124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.508437    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"]
Nov 03 20:04:37.531411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.521755    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-data\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.531411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.521909    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-certs\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.531411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.522082    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-credential-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.531411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.523567    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e98860-af8e-49cb-be50-efe9bb8a8a27-istiod-ca-cert\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.531411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.526721    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-token\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.531411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.527152    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-podinfo\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.531411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.529623    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-envoy\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.532477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.532402    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-socket\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.532776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.532757    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kld42\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-kube-api-access-kld42\") pod \"e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") " pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.602047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.601952    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-kube-api-access-tw65x\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.601995    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.602177    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.602281    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.602327    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.602409    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.602429    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.602531    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.602623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.602589    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.654163 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.654121    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:37.704652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.704611    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.704816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.704699    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.704816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.704742    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.704888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.704800    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.704945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.704897    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.704945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.704930    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705059    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-kube-api-access-tw65x\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705097    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705099    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705066    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705287    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705507    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705657    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.705791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.705728    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.708403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.708155    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.708641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.708605    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.714036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.714014    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-kube-api-access-tw65x\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.714658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.714630    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.802437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.802393    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:37.868277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.864406    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:37.947827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.947798    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"]
Nov 03 20:04:37.948818 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:37.948791    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e98860_af8e_49cb_be50_efe9bb8a8a27.slice/crio-e602eee5cada232aee2ec5c4203011830f30addfff58c1497ceb3eeee5bc91d5 WatchSource:0}: Error finding container e602eee5cada232aee2ec5c4203011830f30addfff58c1497ceb3eeee5bc91d5: Status 404 returned error can't find the container with id e602eee5cada232aee2ec5c4203011830f30addfff58c1497ceb3eeee5bc91d5
Nov 03 20:04:37.955666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.955630    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:37.955754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.955702    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:37.955754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:37.955732    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:38.216806 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.216778    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"]
Nov 03 20:04:38.217484 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:38.217463    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6e5cd9_7b07_4b92_881b_81deaf1524c2.slice/crio-d86635013f34c2704e5f7593d272b3d51ef977cf012002d5220e1e80a7c74117 WatchSource:0}: Error finding container d86635013f34c2704e5f7593d272b3d51ef977cf012002d5220e1e80a7c74117: Status 404 returned error can't find the container with id d86635013f34c2704e5f7593d272b3d51ef977cf012002d5220e1e80a7c74117
Nov 03 20:04:38.220358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.220322    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:38.220441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.220406    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:38.220479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.220453    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:04:38.248791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.248759    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" event={"ID":"93e98860-af8e-49cb-be50-efe9bb8a8a27","Type":"ContainerStarted","Data":"9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9"}
Nov 03 20:04:38.248915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.248819    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" event={"ID":"93e98860-af8e-49cb-be50-efe9bb8a8a27","Type":"ContainerStarted","Data":"e602eee5cada232aee2ec5c4203011830f30addfff58c1497ceb3eeee5bc91d5"}
Nov 03 20:04:38.250139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.250115    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" event={"ID":"dc6e5cd9-7b07-4b92-881b-81deaf1524c2","Type":"ContainerStarted","Data":"d86635013f34c2704e5f7593d272b3d51ef977cf012002d5220e1e80a7c74117"}
Nov 03 20:04:38.277017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.276962    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podStartSLOduration=1.276945135 podStartE2EDuration="1.276945135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.271998775 +0000 UTC m=+1470.941200250" watchObservedRunningTime="2026-11-03 20:04:38.276945135 +0000 UTC m=+1470.946146606"
Nov 03 20:04:38.655230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.655180    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:38.699261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:38.694250    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.257881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.257843    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" event={"ID":"dc6e5cd9-7b07-4b92-881b-81deaf1524c2","Type":"ContainerStarted","Data":"185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd"}
Nov 03 20:04:39.299220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.298852    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podStartSLOduration=2.298833864 podStartE2EDuration="2.298833864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:39.29674785 +0000 UTC m=+1471.965949335" watchObservedRunningTime="2026-11-03 20:04:39.298833864 +0000 UTC m=+1471.968035353"
Nov 03 20:04:39.542535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.542454    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.656205 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.656155    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:39.656403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.656240    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:04:39.657782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.657757    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.739224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.734247    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"]
Nov 03 20:04:39.744075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.743411    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"]
Nov 03 20:04:39.744075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.743521    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.747030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.747006    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b42aea\"/\"distinct-2-w1-b42aea-kserve-self-signed-certs\""
Nov 03 20:04:39.828281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.827570    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:39.863221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.857442    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-model-cache\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.863221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.857588    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-home\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.863221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.857696    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4ln\" (UniqueName: \"kubernetes.io/projected/d3ec540a-467b-4a58-962a-4b804ebfb2ca-kube-api-access-2r4ln\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.863221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.857737    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-dshm\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.863221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.857796    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tls-certs\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.863221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.857958    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tmp-dir\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.902276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.899631    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:39.902276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.899853    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:39.902276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.899905    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.975663    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-home\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.975777    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4ln\" (UniqueName: \"kubernetes.io/projected/d3ec540a-467b-4a58-962a-4b804ebfb2ca-kube-api-access-2r4ln\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.975804    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-dshm\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.975853    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tls-certs\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.976108    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tmp-dir\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.976159    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-model-cache\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.976684    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-model-cache\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.978221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.976920    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-home\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.982225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.979911    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tmp-dir\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:39.987953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.987924    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-dshm\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:40.000318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:39.991843    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tls-certs\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:40.014222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.001123    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4ln\" (UniqueName: \"kubernetes.io/projected/d3ec540a-467b-4a58-962a-4b804ebfb2ca-kube-api-access-2r4ln\") pod \"distinct-2-w1-b42aea-kserve-867c666cd6-67kps\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") " pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:40.059847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.059565    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:40.456403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.456380    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"]
Nov 03 20:04:40.457901 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:04:40.457873    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3ec540a_467b_4a58_962a_4b804ebfb2ca.slice/crio-22b7113ba34c0a42ce21e438436bc477a7458b880f09eef6a30005dc11db1e25 WatchSource:0}: Error finding container 22b7113ba34c0a42ce21e438436bc477a7458b880f09eef6a30005dc11db1e25: Status 404 returned error can't find the container with id 22b7113ba34c0a42ce21e438436bc477a7458b880f09eef6a30005dc11db1e25
Nov 03 20:04:40.654888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.654802    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:40.654888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.654854    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:04:40.704775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.704627    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:40.873641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.873600    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:40.873886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:40.873855    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:04:41.317360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.317251    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" event={"ID":"d3ec540a-467b-4a58-962a-4b804ebfb2ca","Type":"ContainerStarted","Data":"048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed"}
Nov 03 20:04:41.317360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.317314    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" event={"ID":"d3ec540a-467b-4a58-962a-4b804ebfb2ca","Type":"ContainerStarted","Data":"22b7113ba34c0a42ce21e438436bc477a7458b880f09eef6a30005dc11db1e25"}
Nov 03 20:04:41.337381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.337333    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podStartSLOduration=2.337320892 podStartE2EDuration="2.337320892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:41.335623992 +0000 UTC m=+1474.004825464" watchObservedRunningTime="2026-11-03 20:04:41.337320892 +0000 UTC m=+1474.006522363"
Nov 03 20:04:41.658803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.658697    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:41.658803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.658761    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:04:41.806222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.806152    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:41.879221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.873618    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:41.879221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.873683    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:04:41.999220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:41.983614    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:42.040272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.040168    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:42.060372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.060345    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:42.344852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.344777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:42.655191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.655110    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:42.655191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.655162    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:04:42.723179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.723145    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:42.870217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.869666    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:42.870217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:42.869745    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:04:43.655317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:43.655280    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:43.655467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:43.655340    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:04:43.727739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:43.727704    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:43.872225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:43.871767    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:43.872225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:43.871846    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:04:44.655177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:44.655139    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 20:04:44.655384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:44.655224    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 20:04:44.799405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:44.799372    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:44.872741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:44.872703    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 20:04:44.873076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:44.873027    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 20:04:45.621222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:45.620543    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:45.754383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:45.753643    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:45.754383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:45.754164    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:45.756942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:45.756920    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:04:46.091404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:46.091370    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:46.091901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:46.091807    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:46.093234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:46.093168    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:04:46.466280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:46.458786    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.138971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:47.138934    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:47.790656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:47.790625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:48.701839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:48.701793    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:50.061680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:50.061305    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:04:50.881345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:50.881298    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:50.916347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:50.916293    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:50.993355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:50.993298    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.067222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.065279    5282 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.167378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.167277    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:04:51.170003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.169975    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:51.236229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.235512    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:04:51.320226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.319289    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.336260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.335290    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.372228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.361288    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.439290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.433175    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:51.527293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.500188    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:04:51.554220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.550300    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:04:51.569251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:51.569211    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:04:52.054223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.029585    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.292255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.284690    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:52.292255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.094683    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:04:52.292255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.169476    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.332285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.254293    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:04:52.332285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.301648    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:04:52.332285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.301740    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:52.336165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.336115    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.349008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.348968    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.413330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.390316    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.413944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.413915    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:04:52.414319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.414301    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:52.414507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.414493    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:04:52.414805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.414787    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:04:52.495029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.494937    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.598746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.598616    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:04:52.600292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.600253    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.667705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.600885    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:04:52.835590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.835536    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 20:04:52.841573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.841537    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:04:52.881580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.881470    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:52.959790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.959761    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:04:52.968229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:52.961063    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:04:53.079807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.079760    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.287700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.287379    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.288260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.288230    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.354518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.292273    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.381909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.306302    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.407313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.368878    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.407599 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.368940    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.409012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.377292    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.431811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.431758    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.506991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.497581    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.518240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.516630    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:04:53.518240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.516689    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.627042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.618496    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.768266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.767813    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:04:53.896283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.884652    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.970323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.970276    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:53.998222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:53.982574    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Nov 03 20:04:54.306689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.306639    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:04:54.307382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.307348    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:04:54.397389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.395269    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:04:54.397389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.395635    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:04:54.429234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.419596    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:04:54.429234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.419998    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:04:54.522147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.519684    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:04:54.526387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.522287    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:04:54.553821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.553794    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:04:54.555520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.555498    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:04:54.564095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.564028    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:04:54.566491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.566473    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:04:54.631255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.630907    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:04:54.633730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.633702    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:54.775227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.761769    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:04:54.818793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.818713    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:04:54.886178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.886140    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:04:54.890324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.890296    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:04:54.898919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:54.898894    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:04:55.908563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:55.908530    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:56.726298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:56.726267    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.344930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:57.344882    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.479174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:57.479114    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:57.612163 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:57.612068    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:58.216084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:58.216047    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:58.918807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:58.918767    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:04:59.649946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:04:59.649912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:00.072153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:00.072117    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:05:00.083818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:00.083792    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:05:00.399779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:00.399683    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:01.090499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:01.090464    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:01.615699 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:05:01.615660    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:05:01.721405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:01.721367    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:02.401185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:02.401153    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:03.220685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:03.220643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:04.001932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:04.001896    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:04.812734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:04.812703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:05.587377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:05.587332    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:06.410911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:06.410874    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.211338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.211294    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.328343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.328298    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:07.748762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.748729    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"]
Nov 03 20:05:07.753172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.753143    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.772442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.772408    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"]
Nov 03 20:05:07.800875 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.798091    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"]
Nov 03 20:05:07.803892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.803864    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.806845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.806815    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"]
Nov 03 20:05:07.856120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856080    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856145    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856267    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.856347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856308    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.856347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856325    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8sb\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-kube-api-access-2q8sb\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.856536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856355    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856375    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.856536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856510    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856578    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpftg\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-kube-api-access-xpftg\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856688    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.856776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856737    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.856845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856822    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856870    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856942    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.856987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856964    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.856987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856981    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.857054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.856996    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f4e2b4-d910-4268-a01a-ca6e785a6319-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.857084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.857069    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.958532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958492    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpftg\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-kube-api-access-xpftg\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.958741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958558    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.958741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958586    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.958741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958639    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.958741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958689    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.958741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958735    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958761    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958787    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958908    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f4e2b4-d910-4268-a01a-ca6e785a6319-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.958950    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959027    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959090    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959129    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959144    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959181    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959189    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959245    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8sb\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-kube-api-access-2q8sb\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959295    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959323    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959383    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959352    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-credential-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959423    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959451    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959629    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.959872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959699    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f4e2b4-d910-4268-a01a-ca6e785a6319-istiod-ca-cert\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.959872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959742    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-socket\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.960015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959884    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-data\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.960015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.959938    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-certs\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.961927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.961898    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.962048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.962002    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-envoy\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.962263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.962245    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.962332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.962262    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-podinfo\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.966938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.966903    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:07.967068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.967012    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8sb\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-kube-api-access-2q8sb\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.967068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.967058    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-token\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:07.967549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:07.967520    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpftg\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-kube-api-access-xpftg\") pod \"e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") " pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:08.066613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.066528    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:08.119525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.119489    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:08.214245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.209559    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:08.385793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.385552    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"]
Nov 03 20:05:08.397270 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:05:08.396557    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7182a5_3f1e_4ab6_a0fc_aeb5c0b292bc.slice/crio-151cd2a803dd7dd096ba72af91992f22c588717c4f5c323a8b6dc4b09b5ddd60 WatchSource:0}: Error finding container 151cd2a803dd7dd096ba72af91992f22c588717c4f5c323a8b6dc4b09b5ddd60: Status 404 returned error can't find the container with id 151cd2a803dd7dd096ba72af91992f22c588717c4f5c323a8b6dc4b09b5ddd60
Nov 03 20:05:08.400470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.399508    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:08.400470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.399585    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:08.400470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.399616    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:08.409861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.409794    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" event={"ID":"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc","Type":"ContainerStarted","Data":"151cd2a803dd7dd096ba72af91992f22c588717c4f5c323a8b6dc4b09b5ddd60"}
Nov 03 20:05:08.450817 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.450790    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"]
Nov 03 20:05:08.462361 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:05:08.462331    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f4e2b4_d910_4268_a01a_ca6e785a6319.slice/crio-4e3803851bcd5ac03e968cf549d25831bb9acb6416df8cd1945806e16cc49a72 WatchSource:0}: Error finding container 4e3803851bcd5ac03e968cf549d25831bb9acb6416df8cd1945806e16cc49a72: Status 404 returned error can't find the container with id 4e3803851bcd5ac03e968cf549d25831bb9acb6416df8cd1945806e16cc49a72
Nov 03 20:05:08.465472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.465431    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:08.465546 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.465527    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:08.465623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:08.465607    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:09.086222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:09.078679    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:09.432169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:09.432052    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" event={"ID":"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc","Type":"ContainerStarted","Data":"13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05"}
Nov 03 20:05:09.434571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:09.434549    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" event={"ID":"77f4e2b4-d910-4268-a01a-ca6e785a6319","Type":"ContainerStarted","Data":"693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b"}
Nov 03 20:05:09.434732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:09.434719    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" event={"ID":"77f4e2b4-d910-4268-a01a-ca6e785a6319","Type":"ContainerStarted","Data":"4e3803851bcd5ac03e968cf549d25831bb9acb6416df8cd1945806e16cc49a72"}
Nov 03 20:05:09.459399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:09.459346    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podStartSLOduration=2.459331695 podStartE2EDuration="2.459331695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:09.456533935 +0000 UTC m=+1502.125735406" watchObservedRunningTime="2026-11-03 20:05:09.459331695 +0000 UTC m=+1502.128533235"
Nov 03 20:05:09.492854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:09.492786    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podStartSLOduration=2.492764556 podStartE2EDuration="2.492764556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:09.492551866 +0000 UTC m=+1502.161753337" watchObservedRunningTime="2026-11-03 20:05:09.492764556 +0000 UTC m=+1502.161966028"
Nov 03 20:05:10.040738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:10.040700    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:10.098226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:10.090300    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:10.110395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:10.106379    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:05:10.110395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:10.106479    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:05:10.124388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:10.124341    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:10.135227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:10.135171    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:05:10.135714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:10.135681    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:05:11.067412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:11.067372    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:05:11.067803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:11.067439    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:05:11.159238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:11.157166    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:05:11.159238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:11.157246    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:05:11.182358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:11.182060    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:11.960049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:11.960014    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:12.070442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:12.070022    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:05:12.070442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:12.070093    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:05:12.143219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:12.132591    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:05:12.143219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:12.132656    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:05:12.912229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:12.907782    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:13.070229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:13.069549    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:05:13.070229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:13.069607    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:05:13.127230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:13.126573    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:05:13.127230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:13.126634    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:05:13.922225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:13.920494    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:14.068221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:14.067631    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:05:14.068221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:14.067697    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:05:14.124228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:14.123647    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:05:14.124228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:14.123724    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:05:14.794217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:14.790603    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:15.067837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:15.067747    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:05:15.067837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:15.067808    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Nov 03 20:05:15.120374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:15.120333    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:05:15.120580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:15.120404    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Nov 03 20:05:15.650240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:15.649733    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:16.373390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:16.373350    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:16.374147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:16.374124    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:16.375481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:16.375451    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:05:16.417959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:16.417921    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:16.418740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:16.418691    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:16.419994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:16.419966    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:05:16.468146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:16.467471    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:17.153043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:17.152957    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:17.817424 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:17.817378    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:18.693269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:18.693192    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:19.449993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:19.449963    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.153576 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:20.153542    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.278081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:20.278046    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:20.368280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:20.368192    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:05:21.015410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:21.015377    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:22.149221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.139026    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:22.780229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.766301    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:22.780229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.771274    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:22.845229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.845128    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"]
Nov 03 20:05:22.887927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.871422    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"]
Nov 03 20:05:22.887927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.871564    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.889896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.889856    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:22.996640 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.996596    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.996902    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.996949    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.997086    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.997129    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197a300d-62a7-48b6-9b14-0c479de0f8fd-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.997176    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.997256    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.997289    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc86c\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-kube-api-access-qc86c\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.997339    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:22.997989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:22.997602    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.094281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.038438    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.094281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.082234    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100560    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100614    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100649    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100776    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100818    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197a300d-62a7-48b6-9b14-0c479de0f8fd-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100864    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100924    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.100960    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc86c\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-kube-api-access-qc86c\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.101009    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.101652    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-data\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.101936    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-certs\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.102928    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197a300d-62a7-48b6-9b14-0c479de0f8fd-istiod-ca-cert\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.103215    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-credential-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.143222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.120999    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-socket\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.313486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.313430    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 03 20:05:23.314595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.314564    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.400813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.389590    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.400813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.393086    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:05:23.400813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.397695    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.434228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.428553    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.447228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.446903    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc86c\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-kube-api-access-qc86c\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.530223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.525934    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:23.556224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.539856    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-token\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.557180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.557108    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-podinfo\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.591793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.590385    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:05:23.617058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.617020    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-envoy\") pod \"e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") " pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.632468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.632301    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.815236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.799648    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.840218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.817242    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:05:23.903127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.903029    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:23.942232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.938658    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:23.945145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:23.945093    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Nov 03 20:05:24.063625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.063581    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.088503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.088453    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.231285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.222930    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:05:24.231285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.223139    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:05:24.231285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.223291    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:05:24.231285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.223367    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:05:24.231285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.223527    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:05:24.393652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.393592    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.427559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.427524    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:05:24.607343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.580539    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:05:24.941328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.588334    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:24.941328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.921580    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:05:24.941328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.921649    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:05:24.941328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.646352    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:05:24.941328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.646303    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.941328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.939061    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.960284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.683281    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:24.977240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.831713    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:24.977240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.864515    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:05:24.977240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:24.683644    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:25.014238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.001356    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:05:25.014238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.001462    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:05:25.014238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.001504    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:05:25.014238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.001539    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:05:25.123646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.123592    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:25.126106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.126068    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:25.226421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.226294    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b" podUID="95213a91-c3dd-4195-bcea-9e329e10679b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:25.689327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.689246    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:25.924012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.923903    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:05:25.925976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.925941    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:25.951392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.951343    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:05:25.982679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.982627    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:25.989538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.989496    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.023440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.015323    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.046365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.015975    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:26.090484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:25.701711    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 03 20:05:26.091468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.077315    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:05:26.096722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.078291    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:05:26.192887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.192783    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.285191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.285141    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl" podUID="04e563bf-735c-4d94-9de1-d04994490f4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.441221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.364912    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.612389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.612344    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:26.695098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.695055    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:26.953276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.945744    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.360583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.957285    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.365751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:26.957474    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.368313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.089609    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.369679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.114841    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.370035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.127345    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.371246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.127994    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.373234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.128036    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.376396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.128061    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.377990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.223092    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.441791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.441745    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:05:27.709500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.709369    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:27.913618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.913586    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:05:27.913887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.913874    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:05:27.914015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.914005    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:05:27.914134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.914125    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:05:27.914276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.914265    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:05:27.914534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.914523    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:05:27.948381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.943563    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:05:27.975414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.975334    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:05:27.975646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.975632    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:05:27.978699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.978671    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:05:27.995522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:27.995258    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:05:28.003164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.003136    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:05:28.097785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.097755    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"]
Nov 03 20:05:28.109931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.109896    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:28.110129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.110099    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:28.110283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.110154    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:28.299006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.298974    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:28.360977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.360938    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" event={"ID":"197a300d-62a7-48b6-9b14-0c479de0f8fd","Type":"ContainerStarted","Data":"5dbb3e1dd37cd663954cff220639fb00d53a313609c972f24b9c442fbb053117"}
Nov 03 20:05:28.361250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.361227    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" event={"ID":"197a300d-62a7-48b6-9b14-0c479de0f8fd","Type":"ContainerStarted","Data":"67a57abdebfd36af4ad9fce9d6001a41a37aedee3f3982530393ad35a5ea0201"}
Nov 03 20:05:28.440456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.440392    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podStartSLOduration=6.44037577 podStartE2EDuration="6.44037577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:28.438427826 +0000 UTC m=+1521.107629309" watchObservedRunningTime="2026-11-03 20:05:28.44037577 +0000 UTC m=+1521.109577239"
Nov 03 20:05:28.448257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.448213    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:28.938855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:28.938815    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:29.336277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:29.336240    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:29.940168 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:29.940124    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:29.940675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:29.940191    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:05:30.462447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:30.462409    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:30.938954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:30.938920    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:30.939137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:30.938975    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:05:31.299640 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:31.299602    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:31.629854 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:05:31.629756    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:05:31.940073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:31.939976    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:31.940073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:31.940036    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:05:32.037320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:32.037253    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:32.826510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:32.826458    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:32.946815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:32.946778    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:32.947112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:32.947079    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:05:33.654224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:33.653366    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:33.939589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:33.939497    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:33.939589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:33.939567    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:05:34.645095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:34.645047    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:34.939690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:34.939595    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:05:34.939690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:34.939648    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 20:05:35.557223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:35.554002    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:36.192947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:36.192919    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:36.193517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:36.193330    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:36.194110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:36.194092    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:05:36.419678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:36.419625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:37.262509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:37.262473    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:38.325651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:38.325609    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:39.391772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:39.391738    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:40.218255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:40.218214    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:41.080125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:41.080093    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:41.895999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:41.895949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:42.712615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:42.712582    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:43.465847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:43.465784    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.188940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:44.188909    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.594119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:44.594086    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:44.703384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:44.703343    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.044356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:45.044317    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:45.750046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:45.749995    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:46.506793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:46.506764    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:47.388927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:47.388856    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:48.187988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:48.187957    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:48.989820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:48.989788    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:49.867770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:49.866467    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:50.629712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:50.629677    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:51.610837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:51.610737    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.194909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.194875    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"]
Nov 03 20:05:52.201059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.201038    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.211120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.211096    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"]
Nov 03 20:05:52.265151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265108    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhmj\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-kube-api-access-nwhmj\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265176    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265254    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265437    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265479    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265525    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265829    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265932    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.265987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.265961    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.313525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.313494    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"]
Nov 03 20:05:52.318075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.318053    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.335747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.335726    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"]
Nov 03 20:05:52.366558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.366523    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:52.366935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.366914    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.366992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.366952    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.367048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367022    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367082    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367104    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367144    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.367303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367170    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.367303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367189    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367229    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.367303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367263    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367311    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367334    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007ac27c-2a36-426d-bc81-a6c70a55454a-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367405    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.367522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367440    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367468    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckndv\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-kube-api-access-ckndv\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.367522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367477    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.367816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.367772    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.368072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.368035    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.368072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.368065    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.368323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.368086    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.368392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.368352    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.369321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.368543    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhmj\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-kube-api-access-nwhmj\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.369321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.368982    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.370031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.370006    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.370376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.370355    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.376549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.376527    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.377508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.377484    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhmj\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-kube-api-access-nwhmj\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.470119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470022    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470132    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470164    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470220    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470260    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470313    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470345    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007ac27c-2a36-426d-bc81-a6c70a55454a-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470411    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470430    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-data\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470439    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckndv\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-kube-api-access-ckndv\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470566    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470729    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-credential-socket\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.470888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.470838    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-certs\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.471324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.471302    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007ac27c-2a36-426d-bc81-a6c70a55454a-istiod-ca-cert\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.472901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.472877    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-envoy\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.473100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.473084    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-podinfo\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.479174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.479149    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-token\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.479383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.479352    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckndv\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-kube-api-access-ckndv\") pod \"e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") " pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.517552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.517521    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:52.633249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.632644    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:52.942505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:52.925948    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"]
Nov 03 20:05:53.397221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.356529    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:53.398948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.398888    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:53.412384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.399770    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:53.873227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.872818    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:53.873227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.872943    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:53.873227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.872993    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:05:53.898411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.888713    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:53.957062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.952812    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:53.957062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.956183    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:53.957352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.957287    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:05:53.983239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:53.983155    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.115240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.095935    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"]
Nov 03 20:05:54.115240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.096009    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" event={"ID":"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe","Type":"ContainerStarted","Data":"1ae5a8d85fa1bddf94a22cf1c80ea1e38d50e0767dbc3b0fd851736e515ff731"}
Nov 03 20:05:54.115240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.096043    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" event={"ID":"007ac27c-2a36-426d-bc81-a6c70a55454a","Type":"ContainerStarted","Data":"2e101b4fc65f4ed2214d49e2b8a36ea970c2ef511acdfb165f9df414f9816650"}
Nov 03 20:05:54.140702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.140642    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.171914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.171856    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.194923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.192049    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.419251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.392309    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.695268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.693640    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.886993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.886855    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.941408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.941346    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.971490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.971445    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.972093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.972057    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.995245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.973803    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:05:54.995245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.973883    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:05:54.995245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.975497    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.995245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.975783    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:05:54.995245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.989285    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:54.995245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.989630    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:54.995245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:54.989953    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:05:55.203413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.172948    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:05:55.203413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.173774    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:05:55.245473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.237347    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" event={"ID":"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe","Type":"ContainerStarted","Data":"0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f"}
Nov 03 20:05:55.281251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.277612    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" event={"ID":"007ac27c-2a36-426d-bc81-a6c70a55454a","Type":"ContainerStarted","Data":"913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0"}
Nov 03 20:05:55.317235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.309311    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:05:55.317235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.316312    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.325221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.317849    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:05:55.325221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.317916    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:05:55.447890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.403091    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.592218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.567234    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:55.608222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.604312    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:05:55.634238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.626162    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.697775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.670327    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:55.697775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.670394    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:05:55.809803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.728146    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:05:55.809803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.728698    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.809803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.785862    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podStartSLOduration=3.78584348 podStartE2EDuration="3.78584348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:55.492089011 +0000 UTC m=+1548.161290490" watchObservedRunningTime="2026-11-03 20:05:55.78584348 +0000 UTC m=+1548.455044952"
Nov 03 20:05:55.897443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.897342    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.955224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.952344    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:55.979243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.972731    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.034448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.972461    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.034448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.983560    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:05:56.034448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.998291    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.034448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:55.998659    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:05:56.034448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.018726    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podStartSLOduration=4.018704387 podStartE2EDuration="4.018704387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:55.586294341 +0000 UTC m=+1548.255495813" watchObservedRunningTime="2026-11-03 20:05:56.018704387 +0000 UTC m=+1548.687905858"
Nov 03 20:05:56.075535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.047907    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:05:56.075535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.074117    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:05:56.075535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.074184    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:05:56.075535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.074224    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:05:56.075535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.074271    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:05:56.075535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.074317    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:05:56.075535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.074351    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:05:56.177965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.177873    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.311746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.311700    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.363224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.324638    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.582223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.538396    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:05:56.582223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.539517    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:56.582223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.539561    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:05:56.749772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.749725    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:56.823900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.764565    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.824563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.823833    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:05:56.904804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.904696    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:56.977484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:56.974296    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:57.121338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.121271    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:57.505260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.236575    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.505260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.271516    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.505260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.279249    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.505260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.297179    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:57.505260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.297404    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.505260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.447711    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.505260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.463715    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:57.675353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.675307    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:05:57.830428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:57.830314    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:58.360222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.293969    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:58.457222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.449022    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:05:58.457222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.449545    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:05:58.624410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.611231    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:58.624410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.611292    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:05:58.624410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.613150    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:58.624410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.613253    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:05:58.658221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.633687    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:58.659225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.658944    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:05:58.663229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.659356    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:58.663229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:58.659406    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:05:58.760878 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:05:58.700961    5282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.25s"
Nov 03 20:05:59.518799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.518760    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:05:59.519165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.519132    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:05:59.591340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.591295    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.592840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.592806    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.601322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.601281    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.602285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.602255    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:05:59.602710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.602686    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:05:59.607306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.607273    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded"
Nov 03 20:05:59.629223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.629157    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.629848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.629816    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:05:59.634522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.634493    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:05:59.634706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.634682    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:05:59.639304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.639271    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:05:59.639777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.639752    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:05:59.640175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.640153    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 03 20:05:59.640452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.640425    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.664877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.664833    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.665500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.665116    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:05:59.857096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.856980    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:05:59.857741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:05:59.857694    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.230377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.230319    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.402386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.402325    5282 patch_prober.go:29] interesting pod/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:00.403573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.402651    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.440880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.440850    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:06:00.532818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.532717    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:00.533092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.533048    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:00.637217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.615750    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.716532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.637521    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.717185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.637997    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.638040    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.638067    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.639328    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.639359    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.639388    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.640618    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.640655    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.699283    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:06:00.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.699347    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:00.756505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.737710    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:06:00.930316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:00.930043    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:01.007575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.007311    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.065383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.065314    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.450251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.443961    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:01.531245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.523568    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:01.531245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.523652    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:01.766417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.758550    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:06:01.766417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.758627    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:06:01.766720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.766664    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:06:01.793222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:01.768235    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.213318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.196890    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:06:02.226924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.226879    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.232052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.232012    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:02.232969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.232934    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.233790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.233750    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.234446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.234271    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.240896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.240306    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.309538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.241155    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.310301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.241213    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.310833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.265736    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:06:02.311433 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:06:02.281602    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:06:02.444950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.444908    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:06:02.446001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.445978    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:06:02.463560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.463454    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:06:02.469834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.469807    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:06:02.470528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.470508    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:06:02.471663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.471630    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:06:02.471873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.471857    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:06:02.472450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.472426    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:06:02.499844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.494617    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:06:02.542388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.526718    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:02.542932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.542893    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:02.543095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.540879    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:02.543287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.540741    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:06:02.543457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.543445    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:06:02.555819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.544216    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:06:02.634130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.634096    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:06:02.634319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:02.634156    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:06:03.520222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:03.518635    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:03.520222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:03.518695    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:03.633220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:03.633144    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:06:03.633420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:03.633224    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:06:04.356182 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:04.356146    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:04.518612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:04.518573    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:04.518904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:04.518872    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:04.718277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:04.684363    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:06:04.718277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:04.684425    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:06:05.525046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:05.524692    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:05.525046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:05.524743    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:05.647349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:05.647311    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:06:05.647682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:05.647635    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:06:05.999221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:05.996418    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:06.520897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:06.520864    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:06.521166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:06.520913    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:06.635281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:06.634715    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:06:06.635281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:06.634818    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 20:06:07.108070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.108009    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:07.341913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.341864    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"]
Nov 03 20:06:07.347773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.347726    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.371087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.371014    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"]
Nov 03 20:06:07.380068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380038    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380076    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-856bg\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-kube-api-access-856bg\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380280    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380350    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380411    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b78d6-34f2-424e-9ac0-3b9eb3684499-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380479    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380525    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380586    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.380795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.380773    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482369    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-856bg\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-kube-api-access-856bg\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482477    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482530    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482581    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b78d6-34f2-424e-9ac0-3b9eb3684499-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482643    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482686    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482744    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482843    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.482964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482928    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.483018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.482973    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.483308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.483277    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-certs\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.483308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.483302    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-data\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.483503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.483447    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b78d6-34f2-424e-9ac0-3b9eb3684499-istiod-ca-cert\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.483566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.483536    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-credential-socket\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.485476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.485450    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-envoy\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.485782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.485759    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-podinfo\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.496679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.496139    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-856bg\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-kube-api-access-856bg\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.496679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.496635    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-token\") pod \"e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") " pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:07.518609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.518413    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:06:07.518739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.518675    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:06:07.518739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.518725    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 20:06:07.633710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.633454    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:06:07.666626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:07.665235    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:08.011606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.001508    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"]
Nov 03 20:06:08.011606 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:06:08.005095    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266b78d6_34f2_424e_9ac0_3b9eb3684499.slice/crio-32fb875f813b3d4e29d9a168153d0e45585bdca1a516c6f0b948a1653e0daa18 WatchSource:0}: Error finding container 32fb875f813b3d4e29d9a168153d0e45585bdca1a516c6f0b948a1653e0daa18: Status 404 returned error can't find the container with id 32fb875f813b3d4e29d9a168153d0e45585bdca1a516c6f0b948a1653e0daa18
Nov 03 20:06:08.011606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.008519    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:08.011606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.008609    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:08.011606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.008654    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:08.132272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.131703    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:06:08.262575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.262489    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" event={"ID":"266b78d6-34f2-424e-9ac0-3b9eb3684499","Type":"ContainerStarted","Data":"51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3"}
Nov 03 20:06:08.262575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.262539    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" event={"ID":"266b78d6-34f2-424e-9ac0-3b9eb3684499","Type":"ContainerStarted","Data":"32fb875f813b3d4e29d9a168153d0e45585bdca1a516c6f0b948a1653e0daa18"}
Nov 03 20:06:08.263964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.263933    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:06:08.296080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.296022    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podStartSLOduration=1.296008918 podStartE2EDuration="1.296008918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:08.293658318 +0000 UTC m=+1560.962859814" watchObservedRunningTime="2026-11-03 20:06:08.296008918 +0000 UTC m=+1560.965210428"
Nov 03 20:06:08.522788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.522717    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:06:08.523911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.523893    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:06:08.666009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.665971    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:08.786404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:08.786326    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:09.671646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:09.671609    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:09.672172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:09.671666    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:06:09.967285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:09.967104    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:10.666552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:10.666520    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:10.666721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:10.666577    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:06:11.063806 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:11.053877    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:11.666868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:11.666830    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:11.667126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:11.666889    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:06:12.136004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:12.135782    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:12.667100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:12.667064    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:12.667334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:12.667136    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:06:13.279218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:13.268305    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:13.666760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:13.666675    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:13.666760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:13.666741    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:06:14.468220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:14.463706    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:14.666433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:14.666397    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:06:14.666632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:14.666465    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 20:06:15.713219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:15.713089    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.896442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:15.896404    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:15.929241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:15.929212    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:15.929677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:15.929646    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:15.930810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:15.930787    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:06:16.340453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:16.340409    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:16.541731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:16.541643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:17.124629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:17.124599    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:17.846396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:17.846366    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:18.291094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.291062    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:18.344786 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.344754    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"]
Nov 03 20:06:18.349467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.349446    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.358606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.358583    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ea700cd3-openshift-default-dockercfg-fvq54\""
Nov 03 20:06:18.370311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.370286    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"]
Nov 03 20:06:18.580280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.579708    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:18.596220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.588311    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-socket\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.596220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.590970    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-certs\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.596220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.591053    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqn2\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-kube-api-access-zpqn2\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.613421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.613384    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-envoy\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.613684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.613659    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-data\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.613986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.613968    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-token\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.614146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.614132    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-credential-socket\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.614351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.614333    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-podinfo\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.614497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.614481    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istiod-ca-cert\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.719979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.716967    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-socket\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.719979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.717041    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-certs\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.719979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.717098    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqn2\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-kube-api-access-zpqn2\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.719979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.717162    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-envoy\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.724214 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.722638    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-certs\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.724214 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.722902    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-socket\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.761224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.759952    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-data\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.761224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.760179    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-token\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.761224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.760290    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-data\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.761224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.760307    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-credential-socket\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.761224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.760433    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-podinfo\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.761224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.760506    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istiod-ca-cert\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.761224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.760566    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-credential-socket\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.762285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.762253    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istiod-ca-cert\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.785284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.771000    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-podinfo\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.792928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.792574    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:18.808221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.803554    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-envoy\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.808221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.804414    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqn2\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-kube-api-access-zpqn2\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.832270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.831587    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-token\") pod \"e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") " pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.964671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.964633    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:18.983257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:18.983224    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.373223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:19.368382    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"]
Nov 03 20:06:19.448223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:19.391350    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:19.448223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:19.391455    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:19.448223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:19.391506    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:06:19.689236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:19.688756    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:19.737736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:19.735925    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq" event={"ID":"d1255c5a-19f1-44e2-a8a7-70d0da40b018","Type":"ContainerStarted","Data":"fd5e93c5065be3b4efc13dcff7aa05e207b29c54fa42fcfd5b54be795daf5ab7"}
Nov 03 20:06:19.928465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:19.918614    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.166184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:20.166118    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.325533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:20.325497    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:20.747403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:20.747365    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq" event={"ID":"d1255c5a-19f1-44e2-a8a7-70d0da40b018","Type":"ContainerStarted","Data":"3a68469c1c5b05783badae65fbcc53b9f20b8a7eaba305cc21937a52b5c09855"}
Nov 03 20:06:20.803168 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:20.803122    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq" podStartSLOduration=2.803108403 podStartE2EDuration="2.803108403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:20.799836942 +0000 UTC m=+1573.469038449" watchObservedRunningTime="2026-11-03 20:06:20.803108403 +0000 UTC m=+1573.472309873"
Nov 03 20:06:20.965066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:20.965022    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:20.970759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:20.970734    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:21.104923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.104851    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:21.148067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.148030    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"]
Nov 03 20:06:21.153463 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.153434    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.160087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.160058    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8fdab3c3\""
Nov 03 20:06:21.160292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.160274    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8fdab3c3-dockercfg-l46xr\""
Nov 03 20:06:21.163871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.163843    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"]
Nov 03 20:06:21.258188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.258146    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.272234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.266020    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8w6p\" (UniqueName: \"kubernetes.io/projected/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-kube-api-access-n8w6p\") pod \"payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.282230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.280145    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:21.379273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.378359    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8w6p\" (UniqueName: \"kubernetes.io/projected/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-kube-api-access-n8w6p\") pod \"payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.379273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.378743    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.383222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.379778    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.439770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.431230    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8w6p\" (UniqueName: \"kubernetes.io/projected/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-kube-api-access-n8w6p\") pod \"payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.466257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.456995    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:21.487222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.481698    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:21.787393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.769834    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:21.787393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:21.771251    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:06:22.109672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:22.109628    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"]
Nov 03 20:06:22.112651 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:06:22.112621    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1602b7_4cbd_4084_a6b8_5a48c84d1979.slice/crio-e73a1bff4359ffc23c5ce9c202d841b619651c44868a738fc0571b48a70b7ee0 WatchSource:0}: Error finding container e73a1bff4359ffc23c5ce9c202d841b619651c44868a738fc0571b48a70b7ee0: Status 404 returned error can't find the container with id e73a1bff4359ffc23c5ce9c202d841b619651c44868a738fc0571b48a70b7ee0
Nov 03 20:06:22.821095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:22.821055    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:22.824696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:22.824668    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k" event={"ID":"eb1602b7-4cbd-4084-a6b8-5a48c84d1979","Type":"ContainerStarted","Data":"86e2027ec961d1a111d237b77c6133d259112fbf2e7cf5c7d127151971fe1172"}
Nov 03 20:06:22.824860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:22.824761    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k" event={"ID":"eb1602b7-4cbd-4084-a6b8-5a48c84d1979","Type":"ContainerStarted","Data":"e73a1bff4359ffc23c5ce9c202d841b619651c44868a738fc0571b48a70b7ee0"}
Nov 03 20:06:22.824860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:22.824819    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:22.891221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:22.890909    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k" podStartSLOduration=1.890888288 podStartE2EDuration="1.890888288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:22.873654353 +0000 UTC m=+1575.542855825" watchObservedRunningTime="2026-11-03 20:06:22.890888288 +0000 UTC m=+1575.560089762"
Nov 03 20:06:23.033763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:23.033725    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:23.268563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:23.268513    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:24.261625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:24.261587    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:24.519269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:24.518703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.606742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.602536    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.715364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.715331    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"]
Nov 03 20:06:25.720011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.719990    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:25.730356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.730335    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1cca2dba-dockercfg-sd7v4\""
Nov 03 20:06:25.731662 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.731645    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1cca2dba\""
Nov 03 20:06:25.743484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.743457    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"]
Nov 03 20:06:25.747376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.747353    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"]
Nov 03 20:06:25.752420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.752396    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:25.769222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.769173    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"]
Nov 03 20:06:25.771393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.771368    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02059338-a95a-4f71-9e66-8d9b4e6e6646-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:25.771869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.771843    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvqlh\" (UniqueName: \"kubernetes.io/projected/02059338-a95a-4f71-9e66-8d9b4e6e6646-kube-api-access-qvqlh\") pod \"payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:25.900275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.861357    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:25.984221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.901749    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02059338-a95a-4f71-9e66-8d9b4e6e6646-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:25.984221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.976646    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c616662-7a72-4200-b53e-533f3b6a21f0-plugins-config-volume\") pod \"payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:25.984221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.976963    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvqlh\" (UniqueName: \"kubernetes.io/projected/02059338-a95a-4f71-9e66-8d9b4e6e6646-kube-api-access-qvqlh\") pod \"payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:25.984221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.977088    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhjz\" (UniqueName: \"kubernetes.io/projected/3c616662-7a72-4200-b53e-533f3b6a21f0-kube-api-access-mmhjz\") pod \"payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:25.984221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:25.902744    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02059338-a95a-4f71-9e66-8d9b4e6e6646-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:26.019222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.008035    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvqlh\" (UniqueName: \"kubernetes.io/projected/02059338-a95a-4f71-9e66-8d9b4e6e6646-kube-api-access-qvqlh\") pod \"payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:26.037021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.035794    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:26.085098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.079745    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhjz\" (UniqueName: \"kubernetes.io/projected/3c616662-7a72-4200-b53e-533f3b6a21f0-kube-api-access-mmhjz\") pod \"payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:26.085098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.079869    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c616662-7a72-4200-b53e-533f3b6a21f0-plugins-config-volume\") pod \"payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:26.085098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.080913    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c616662-7a72-4200-b53e-533f3b6a21f0-plugins-config-volume\") pod \"payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:26.131232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.127869    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhjz\" (UniqueName: \"kubernetes.io/projected/3c616662-7a72-4200-b53e-533f3b6a21f0-kube-api-access-mmhjz\") pod \"payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") " pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:26.413331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.382394    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:26.620684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.620271    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"]
Nov 03 20:06:26.817940 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:06:26.817895    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c616662_7a72_4200_b53e_533f3b6a21f0.slice/crio-983228d8e308d373e22092e1a03c974f8119e35ca945646d3f9d39da8fa124bc WatchSource:0}: Error finding container 983228d8e308d373e22092e1a03c974f8119e35ca945646d3f9d39da8fa124bc: Status 404 returned error can't find the container with id 983228d8e308d373e22092e1a03c974f8119e35ca945646d3f9d39da8fa124bc
Nov 03 20:06:26.818144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.818121    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"]
Nov 03 20:06:26.950960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:26.950785    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:27.085169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.085131    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r" event={"ID":"3c616662-7a72-4200-b53e-533f3b6a21f0","Type":"ContainerStarted","Data":"2c42a94616d664536f3a0988784f37fb8770ea84326bcb0cbf13b2450c9466bb"}
Nov 03 20:06:27.085372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.085192    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r" event={"ID":"3c616662-7a72-4200-b53e-533f3b6a21f0","Type":"ContainerStarted","Data":"983228d8e308d373e22092e1a03c974f8119e35ca945646d3f9d39da8fa124bc"}
Nov 03 20:06:27.085372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.085360    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:27.086833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.086808    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79" event={"ID":"02059338-a95a-4f71-9e66-8d9b4e6e6646","Type":"ContainerStarted","Data":"f0eab9a752fff38ce3ebbfe66c51534a9e6b9964b9006f8d1b9b87a6048aa04d"}
Nov 03 20:06:27.086952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.086848    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79" event={"ID":"02059338-a95a-4f71-9e66-8d9b4e6e6646","Type":"ContainerStarted","Data":"db47f2e5dc01af66c495f9c755578bc85d4085fb239887231394582d390ae267"}
Nov 03 20:06:27.086952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.086896    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:27.114999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.114951    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r" podStartSLOduration=2.114932185 podStartE2EDuration="2.114932185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:27.109974606 +0000 UTC m=+1579.779176076" watchObservedRunningTime="2026-11-03 20:06:27.114932185 +0000 UTC m=+1579.784133653"
Nov 03 20:06:27.130631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.130591    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79" podStartSLOduration=2.130577385 podStartE2EDuration="2.130577385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:27.129504111 +0000 UTC m=+1579.798705582" watchObservedRunningTime="2026-11-03 20:06:27.130577385 +0000 UTC m=+1579.799778857"
Nov 03 20:06:27.894790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:27.894734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:28.627335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:28.627298    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:29.295883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:29.295847    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:29.436864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:29.436741    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:29.569670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:29.569589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:30.121232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:30.121172    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:30.973229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:30.966033    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:31.090305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:31.090273    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:06:31.615878 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:06:31.615843    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:06:31.817088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:31.817054    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:31.944146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:31.944058    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:32.080028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:32.079999    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:33.479257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.471038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:33.488378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.488327    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.558416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.558370    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.583706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.583656    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:06:33.691224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.643247    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:33.743813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.743694    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:06:33.772717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.771494    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.773931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.771590    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.921268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.845029    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.921268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.863304    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.934393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.925056    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:33.994727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:33.989398    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.020952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.020905    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.266278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.257328    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:06:34.409624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.409592    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"]
Nov 03 20:06:34.419951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.419917    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"]
Nov 03 20:06:34.420304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.420276    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.464165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.463643    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1cca2dba-serving-cert\""
Nov 03 20:06:34.501180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.501129    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.502989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.502943    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:06:34.529631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.517379    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:06:34.534483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.534408    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqk6n\" (UniqueName: \"kubernetes.io/projected/5710754b-9d46-4eb8-a3bf-032b7268e2e0-kube-api-access-tqk6n\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.535144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.535027    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5710754b-9d46-4eb8-a3bf-032b7268e2e0-openshift-service-ca\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.535636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.535518    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.535894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.535874    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-metrics-certs\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.576583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.576537    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.578113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.577683    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:06:34.594116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.586918    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.594116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.587473    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:06:34.700226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.692337    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5710754b-9d46-4eb8-a3bf-032b7268e2e0-openshift-service-ca\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.710229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.701054    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.711042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.711008    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-metrics-certs\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.718223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.711620    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqk6n\" (UniqueName: \"kubernetes.io/projected/5710754b-9d46-4eb8-a3bf-032b7268e2e0-kube-api-access-tqk6n\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.718650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.699482    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5710754b-9d46-4eb8-a3bf-032b7268e2e0-openshift-service-ca\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.726231 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:06:34.719599    5282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-serving-cert: secret "maas-api-e2e-derive-1cca2dba-serving-cert" not found
Nov 03 20:06:34.726231 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:06:34.719667    5282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls podName:5710754b-9d46-4eb8-a3bf-032b7268e2e0 nodeName:}" failed. No retries permitted until 2026-11-03 20:06:35.219645968 +0000 UTC m=+1587.888847417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls") pod "maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" (UID: "5710754b-9d46-4eb8-a3bf-032b7268e2e0") : secret "maas-api-e2e-derive-1cca2dba-serving-cert" not found
Nov 03 20:06:34.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.820975    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-metrics-certs\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:34.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.821566    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:34.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.821849    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.824271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.822092    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:34.825090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.824777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:06:34.825090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.824840    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:06:34.825090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.824882    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:06:34.992227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.947483    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.992227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.964816    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:06:34.992227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.964872    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:34.992227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.965307    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:06:34.992227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:34.991949    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqk6n\" (UniqueName: \"kubernetes.io/projected/5710754b-9d46-4eb8-a3bf-032b7268e2e0-kube-api-access-tqk6n\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:35.204269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.006619    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:35.204269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.007046    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:06:35.204269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.059458    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:35.204269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.073479    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:06:35.204269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.175320    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:06:35.255416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.228015    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:35.256213 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:06:35.231334    5282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-serving-cert: secret "maas-api-e2e-derive-1cca2dba-serving-cert" not found
Nov 03 20:06:35.256213 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:06:35.255814    5282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls podName:5710754b-9d46-4eb8-a3bf-032b7268e2e0 nodeName:}" failed. No retries permitted until 2026-11-03 20:06:36.255784129 +0000 UTC m=+1588.924985589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls") pod "maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" (UID: "5710754b-9d46-4eb8-a3bf-032b7268e2e0") : secret "maas-api-e2e-derive-1cca2dba-serving-cert" not found
Nov 03 20:06:35.288223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.287356    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:06:35.289369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.289342    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:06:35.540312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.512805    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:06:35.540312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.513150    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:06:35.630223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.589905    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:06:36.037274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.607910    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:06:36.037274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.742159    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.037274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.920382    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.038322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.920425    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.040332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.925575    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.041605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.967337    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.056233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.967337    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.056233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:35.969049    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:36.056233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.013315    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.074226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.066806    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.111369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.111332    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:06:36.291841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.291743    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.295843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.295792    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:36.520292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.520231    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:36.544996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.544738    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:06:36.558920 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.558838    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\") pod \"maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:36.648132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:36.648095    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:37.061404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.039858    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.061404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.040632    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.061404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.041291    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.061404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.041578    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.061404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.042282    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.061990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.061951    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.070218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.062426    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:37.075219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.070781    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.104357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.104294    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:37.300261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.293872    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:37.314247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.311415    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:37.314247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.311484    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:37.560290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.559868    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:06:37.597238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:37.563619    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.047747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.047698    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.048416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.048383    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:38.048884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.048861    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:06:38.049268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.049246    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:06:38.049947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.049920    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:38.080220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.076344    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.098369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.076567    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.123928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.111447    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.191680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.112347    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.238432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.238378    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.311067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.308986    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.353945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.353906    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:06:38.816282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.812597    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:06:38.816282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.601470    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:38.920230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.642521    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 03 20:06:38.924225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.704326    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:38.947223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:38.609558    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.126137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.108959    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.202644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.199338    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.217853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.217810    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.218298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.218268    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.219999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.219975    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded"
Nov 03 20:06:39.220146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.220121    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.220384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.220362    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.221464 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.221441    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.224189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.224152    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.229869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.225028    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:06:39.300233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.292838    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:39.332676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.321344    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.362229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.345616    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:39.362560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.362523    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.785569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.781259    5282 patch_prober.go:29] interesting pod/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:06:39.785569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.781349    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:39.875269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:39.866548    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.086222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.082599    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:06:40.220209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.220146    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b" podUID="95213a91-c3dd-4195-bcea-9e329e10679b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Nov 03 20:06:40.223368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.223334    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.223539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.221160    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.223650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.220461    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.247110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.221708    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.249593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.221729    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:06:40.258603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.223723    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:06:40.288297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.221681    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.359281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.234584    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:40.362228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.247459    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.362228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.247674    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 03 20:06:40.362228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.247700    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb" podUID="3bf6a00b-f4b7-4909-8155-e849fecd35ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": context deadline exceeded"
Nov 03 20:06:40.362228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.247724    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.362228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.361174    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:40.581025 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:06:40.580972    5282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.11s"
Nov 03 20:06:40.581289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.581272    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:06:40.977494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:40.977443    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.229098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.228989    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.246510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.239667    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Nov 03 20:06:41.277223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.273614    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.305214 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.303707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:41.307507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.305501    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.307741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.305527    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.309511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.309491    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"]
Nov 03 20:06:41.319289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.319228    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.384476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.384410    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.421568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.421529    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:06:41.440227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.435651    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:06:41.440227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.436523    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:06:41.468252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.451360    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:06:41.562831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.560189    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:06:41.635708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.635663    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:06:41.827183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.827151    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:06:41.827394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.827226    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:06:41.829546 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.829386    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:41.840460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.840436    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:06:41.844836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.844817    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:06:41.865744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.865715    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" event={"ID":"5710754b-9d46-4eb8-a3bf-032b7268e2e0","Type":"ContainerStarted","Data":"cbf3507ec71fd9d85c12d69fd98cbdf3e4eaaa23b35720af2262a1e26514649e"}
Nov 03 20:06:41.922985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.922956    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:06:41.924886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.924864    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:06:41.950647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.950622    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:06:41.960230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.960070    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:06:41.985611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.985570    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:06:41.986243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:41.986224    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:06:42.115897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:42.114299    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:42.524308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:42.524265    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:42.874771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:42.874681    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" event={"ID":"5710754b-9d46-4eb8-a3bf-032b7268e2e0","Type":"ContainerStarted","Data":"8c972fcb4fa9de6ac1d2afeb24cd7299a3abe27ce2259b46b7f651554cdceb9a"}
Nov 03 20:06:42.874914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:42.874870    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:42.899336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:42.899285    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" podStartSLOduration=8.530249796 podStartE2EDuration="9.899271872s" totalImagesPullingTime="1.369022076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:42.896141872 +0000 UTC m=+1595.565343344" watchObservedRunningTime="2026-11-03 20:06:42.899271872 +0000 UTC m=+1595.568473342"
Nov 03 20:06:43.275740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:43.275705    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:44.174227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:44.170744    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:44.904907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:44.904864    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.097424 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:45.080314    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:45.805057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:45.805023    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:46.860221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:46.859697    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:47.804393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:47.804363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:48.614577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:48.614544    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:48.847828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:48.847791    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:48.887183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:48.887098    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:06:49.127931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:49.127896    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.522424 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:49.522394    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.686339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:49.686292    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:49.833853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:49.833728    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:50.669993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:50.666963    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:50.815928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:50.815894    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:50.951578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:50.951485    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:51.940911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:51.940876    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.278070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:52.278025    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:52.922238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:52.922177    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:53.967926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:53.967839    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:55.107176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:55.107138    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:56.329137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:56.329096    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:57.630832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:57.630798    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:58.702975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:58.702935    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:06:59.479967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:06:59.479918    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.301029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:00.300992    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:00.870713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:00.870673    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"]
Nov 03 20:07:00.962701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:00.962661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.093921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:01.093870    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.272237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:01.272107    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.421586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:01.421556    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.543119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:01.543041    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:01.616437 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:01.616398    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:07:01.981501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:01.981322    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:02.764475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:02.764436    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:02.986069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:02.986036    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:03.035938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:03.035853    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"]
Nov 03 20:07:03.080326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:03.079664    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"]
Nov 03 20:07:03.155572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:03.155512    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:03.505220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:03.501775    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:03.783520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:03.783432    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.343421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:04.343366    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.452256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:04.452222    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:04.600748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:04.600657    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.395711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:05.395676    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:05.873366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:05.873289    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k" podUID="eb1602b7-4cbd-4084-a6b8-5a48c84d1979" containerName="payload-pre-processing" containerID="cri-o://86e2027ec961d1a111d237b77c6133d259112fbf2e7cf5c7d127151971fe1172" gracePeriod=25
Nov 03 20:07:06.303757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.303718    5282 generic.go:356] "Generic (PLEG): container finished" podID="eb1602b7-4cbd-4084-a6b8-5a48c84d1979" containerID="86e2027ec961d1a111d237b77c6133d259112fbf2e7cf5c7d127151971fe1172" exitCode=0
Nov 03 20:07:06.303936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.303853    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k" event={"ID":"eb1602b7-4cbd-4084-a6b8-5a48c84d1979","Type":"ContainerDied","Data":"86e2027ec961d1a111d237b77c6133d259112fbf2e7cf5c7d127151971fe1172"}
Nov 03 20:07:06.340952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.340916    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:06.478270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.477589    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:07:06.554858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.554771    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-plugins-config-volume\") pod \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") "
Nov 03 20:07:06.555116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.555098    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-kube-api-access-n8w6p\" (UniqueName: \"kubernetes.io/projected/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-kube-api-access-n8w6p\") pod \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\" (UID: \"eb1602b7-4cbd-4084-a6b8-5a48c84d1979\") "
Nov 03 20:07:06.555226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.555160    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-plugins-config-volume" pod "eb1602b7-4cbd-4084-a6b8-5a48c84d1979" (UID: "eb1602b7-4cbd-4084-a6b8-5a48c84d1979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:06.555721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.555702    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:06.557340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.557318    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-kube-api-access-n8w6p" pod "eb1602b7-4cbd-4084-a6b8-5a48c84d1979" (UID: "eb1602b7-4cbd-4084-a6b8-5a48c84d1979"). InnerVolumeSpecName "kube-api-access-n8w6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:06.656833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:06.656798    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8w6p\" (UniqueName: \"kubernetes.io/projected/eb1602b7-4cbd-4084-a6b8-5a48c84d1979-kube-api-access-n8w6p\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:07.342221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.341724    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k" event={"ID":"eb1602b7-4cbd-4084-a6b8-5a48c84d1979","Type":"ContainerDied","Data":"e73a1bff4359ffc23c5ce9c202d841b619651c44868a738fc0571b48a70b7ee0"}
Nov 03 20:07:07.342221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.341799    5282 scope.go:122] "RemoveContainer" containerID="86e2027ec961d1a111d237b77c6133d259112fbf2e7cf5c7d127151971fe1172"
Nov 03 20:07:07.342221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.341949    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"
Nov 03 20:07:07.346934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.346909    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.377336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.377268    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"]
Nov 03 20:07:07.380767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.380745    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8fdab3c3-f7f599547-ttj2k"]
Nov 03 20:07:07.459181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.459143    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1602b7-4cbd-4084-a6b8-5a48c84d1979" path="/var/lib/kubelet/pods/eb1602b7-4cbd-4084-a6b8-5a48c84d1979/volumes"
Nov 03 20:07:07.589262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.589183    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.724116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.724052    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:07.863299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.863264    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.036280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:07.991750    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.102220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:08.078687    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r" podUID="3c616662-7a72-4200-b53e-533f3b6a21f0" containerName="payload-processing" containerID="cri-o://2c42a94616d664536f3a0988784f37fb8770ea84326bcb0cbf13b2450c9466bb" gracePeriod=25
Nov 03 20:07:08.102220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:08.082435    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79" podUID="02059338-a95a-4f71-9e66-8d9b4e6e6646" containerName="payload-pre-processing" containerID="cri-o://f0eab9a752fff38ce3ebbfe66c51534a9e6b9964b9006f8d1b9b87a6048aa04d" gracePeriod=25
Nov 03 20:07:08.232320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:08.225712    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79" podUID="02059338-a95a-4f71-9e66-8d9b4e6e6646" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:07:08.467765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:08.467682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.758220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:08.753869    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:08.903270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:08.892760    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.046228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.045419    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.059223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.058551    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.059223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.059136    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:07:09.063239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.059479    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded"
Nov 03 20:07:09.089223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.076637    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.089223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.077156    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.089223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.080368    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.114239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.093165    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:07:09.114239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.094554    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.114239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.094855    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:07:09.230280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.228317    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.354223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.296311    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.354223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.340726    5282 patch_prober.go:29] interesting pod/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:07:09.354223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.340797    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.608280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.544554    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:09.672588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.654715    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:07:09.791829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.749561    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:09.825254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:09.817923    5282 generic.go:356] "Generic (PLEG): container finished" podID="3c616662-7a72-4200-b53e-533f3b6a21f0" containerID="2c42a94616d664536f3a0988784f37fb8770ea84326bcb0cbf13b2450c9466bb" exitCode=0
Nov 03 20:07:10.120445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.120361    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.465587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.152623    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.486223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.193055    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.507539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.193106    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.519273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.193130    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.531270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.210716    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.531270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.342287    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.531270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.345883    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.531587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.387837    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.549561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.549519    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:10.559328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.396802    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.562676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.392274    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.563727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.557956    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:10.604983    5282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.136s"
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.608476    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r" event={"ID":"3c616662-7a72-4200-b53e-533f3b6a21f0","Type":"ContainerDied","Data":"2c42a94616d664536f3a0988784f37fb8770ea84326bcb0cbf13b2450c9466bb"}
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.608791    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.608858    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.608880    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"]
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.613643    5282 cpu_manager.go:414] "containerMap: removing container" podUID="eb1602b7-4cbd-4084-a6b8-5a48c84d1979" containerName="payload-pre-processing"
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.613666    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1602b7-4cbd-4084-a6b8-5a48c84d1979" containerName="payload-pre-processing"
Nov 03 20:07:10.614222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.613877    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1602b7-4cbd-4084-a6b8-5a48c84d1979" containerName="payload-pre-processing"
Nov 03 20:07:10.659887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.659838    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:10.662789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.662758    5282 generic.go:356] "Generic (PLEG): container finished" podID="02059338-a95a-4f71-9e66-8d9b4e6e6646" containerID="f0eab9a752fff38ce3ebbfe66c51534a9e6b9964b9006f8d1b9b87a6048aa04d" exitCode=0
Nov 03 20:07:10.960540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.960512    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"]
Nov 03 20:07:10.972376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.972320    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79" event={"ID":"02059338-a95a-4f71-9e66-8d9b4e6e6646","Type":"ContainerDied","Data":"f0eab9a752fff38ce3ebbfe66c51534a9e6b9964b9006f8d1b9b87a6048aa04d"}
Nov 03 20:07:10.984221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.980345    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:07:11.007648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.007612    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:07:11.007942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.007922    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:07:11.008102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.008087    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:07:11.019896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.012294    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:07:11.123388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.111589    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:07:11.123388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.111646    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:07:11.123388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.111686    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:07:11.123388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.111722    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:07:11.123388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.111756    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:07:11.123388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:10.961064    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.123388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.100035    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:07:11.159223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.145180    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c13f7966-gw-openshift-default-dockercfg-2kc88\""
Nov 03 20:07:11.197282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.152529    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:11.197628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.160467    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-data\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.197893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.197872    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-podinfo\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.198055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.198039    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-socket\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.198271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.198253    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-credential-socket\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.198479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.198458    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-certs\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.198605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.198588    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-envoy\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.199013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.198993    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3620eb-8848-4b9b-a1bd-085897d9a635-istiod-ca-cert\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.199162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.199144    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4xlp\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-kube-api-access-w4xlp\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.205224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.199485    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-token\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.309853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.301403    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.380278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.337901    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c616662-7a72-4200-b53e-533f3b6a21f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c616662-7a72-4200-b53e-533f3b6a21f0-plugins-config-volume\") pod \"3c616662-7a72-4200-b53e-533f3b6a21f0\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") "
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.419881    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c616662-7a72-4200-b53e-533f3b6a21f0-kube-api-access-mmhjz\" (UniqueName: \"kubernetes.io/projected/3c616662-7a72-4200-b53e-533f3b6a21f0-kube-api-access-mmhjz\") pod \"3c616662-7a72-4200-b53e-533f3b6a21f0\" (UID: \"3c616662-7a72-4200-b53e-533f3b6a21f0\") "
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420253    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-data\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420341    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-podinfo\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420384    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-socket\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420452    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-credential-socket\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420536    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-certs\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420565    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-envoy\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420760    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3620eb-8848-4b9b-a1bd-085897d9a635-istiod-ca-cert\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420809    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xlp\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-kube-api-access-w4xlp\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.420938    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-token\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.449224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.352987    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c616662-7a72-4200-b53e-533f3b6a21f0-plugins-config-volume" pod "3c616662-7a72-4200-b53e-533f3b6a21f0" (UID: "3c616662-7a72-4200-b53e-533f3b6a21f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:11.489209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.404229    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 03 20:07:11.539312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.508106    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-data\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.539312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.508556    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-socket\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.539312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.529953    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.539312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.538451    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.539312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.538837    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.539312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.539119    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.539787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.539412    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.539787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.539690    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.583162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.547521    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-credential-socket\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.583162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.550322    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.583162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.550750    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.631511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.631408    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3620eb-8848-4b9b-a1bd-085897d9a635-istiod-ca-cert\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.633008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.632983    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-certs\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.654392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.654346    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.733223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.678160    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.733223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.683443    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:11.733223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.693821    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:11.736457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.736421    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-podinfo\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.807371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.802144    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c616662-7a72-4200-b53e-533f3b6a21f0-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:11.837224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.815414    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-envoy\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:11.912533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:11.912487    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xlp\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-kube-api-access-w4xlp\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:12.059669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.059627    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c616662-7a72-4200-b53e-533f3b6a21f0-kube-api-access-mmhjz" pod "3c616662-7a72-4200-b53e-533f3b6a21f0" (UID: "3c616662-7a72-4200-b53e-533f3b6a21f0"). InnerVolumeSpecName "kube-api-access-mmhjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:12.265602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.063867    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:12.269262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.071817    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-token\") pod \"e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") " pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:12.470247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.450115    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:12.477535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.476315    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmhjz\" (UniqueName: \"kubernetes.io/projected/3c616662-7a72-4200-b53e-533f3b6a21f0-kube-api-access-mmhjz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:12.595223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.582484    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.602790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.602755    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"
Nov 03 20:07:12.641838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.611664    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.641838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.611817    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.641838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.629313    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r" event={"ID":"3c616662-7a72-4200-b53e-533f3b6a21f0","Type":"ContainerDied","Data":"983228d8e308d373e22092e1a03c974f8119e35ca945646d3f9d39da8fa124bc"}
Nov 03 20:07:12.641838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.629379    5282 scope.go:122] "RemoveContainer" containerID="2c42a94616d664536f3a0988784f37fb8770ea84326bcb0cbf13b2450c9466bb"
Nov 03 20:07:12.685962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.685914    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.687342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.687309    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.703412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.703371    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.748391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.748316    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.796478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.749656    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.812028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.688355    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.812552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.751425    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.813975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.751455    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.815062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.775289    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:12.832403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.722138    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:07:12.849222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.834659    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:07:13.026234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:12.994309    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.026234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.001310    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02059338-a95a-4f71-9e66-8d9b4e6e6646-kube-api-access-qvqlh\" (UniqueName: \"kubernetes.io/projected/02059338-a95a-4f71-9e66-8d9b4e6e6646-kube-api-access-qvqlh\") pod \"02059338-a95a-4f71-9e66-8d9b4e6e6646\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") "
Nov 03 20:07:13.026234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.001473    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02059338-a95a-4f71-9e66-8d9b4e6e6646-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02059338-a95a-4f71-9e66-8d9b4e6e6646-plugins-config-volume\") pod \"02059338-a95a-4f71-9e66-8d9b4e6e6646\" (UID: \"02059338-a95a-4f71-9e66-8d9b4e6e6646\") "
Nov 03 20:07:13.026234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.003577    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02059338-a95a-4f71-9e66-8d9b4e6e6646-plugins-config-volume" pod "02059338-a95a-4f71-9e66-8d9b4e6e6646" (UID: "02059338-a95a-4f71-9e66-8d9b4e6e6646"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:13.137223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.136334    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02059338-a95a-4f71-9e66-8d9b4e6e6646-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:13.196861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.195834    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02059338-a95a-4f71-9e66-8d9b4e6e6646-kube-api-access-qvqlh" pod "02059338-a95a-4f71-9e66-8d9b4e6e6646" (UID: "02059338-a95a-4f71-9e66-8d9b4e6e6646"). InnerVolumeSpecName "kube-api-access-qvqlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:13.252238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.245144    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvqlh\" (UniqueName: \"kubernetes.io/projected/02059338-a95a-4f71-9e66-8d9b4e6e6646-kube-api-access-qvqlh\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:13.530249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.503504    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.680922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.680773    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:13.739260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.698686    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:13.739260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.716396    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:07:13.739260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.716439    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.756392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.756346    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:07:13.871269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.794304    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.871269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.807670    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:13.871269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.846310    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:07:13.874223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.874153    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.876377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.876341    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.876856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.876828    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.896284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.881213    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:13.897626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:13.896024    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.023638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.023604    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:07:14.032897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.032849    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79" event={"ID":"02059338-a95a-4f71-9e66-8d9b4e6e6646","Type":"ContainerDied","Data":"db47f2e5dc01af66c495f9c755578bc85d4085fb239887231394582d390ae267"}
Nov 03 20:07:14.033072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.032939    5282 scope.go:122] "RemoveContainer" containerID="f0eab9a752fff38ce3ebbfe66c51534a9e6b9964b9006f8d1b9b87a6048aa04d"
Nov 03 20:07:14.033160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.033144    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"
Nov 03 20:07:14.657221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.656921    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:14.760408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.760361    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.859074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.850648    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:14.859074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.850828    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:14.859074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.850936    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:14.859074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.852238    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.859074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.853561    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:14.859074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.854680    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:14.868031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.867880    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.870229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.870167    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.871931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.870180    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:14.893224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.886003    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:07:14.897720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.897684    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:14.926233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.924842    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:14.950249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:14.949110    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.136551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.106002    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl" podUID="04e563bf-735c-4d94-9de1-d04994490f4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.137070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.137031    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b" podUID="95213a91-c3dd-4195-bcea-9e329e10679b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/ready\": context deadline exceeded"
Nov 03 20:07:15.192725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.192670    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-44wlb" podUID="3bf6a00b-f4b7-4909-8155-e849fecd35ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.314226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.310437    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:07:15.410182 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.410155    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:07:15.586325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.586296    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:07:15.757535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.757495    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm" event={"ID":"3f3620eb-8848-4b9b-a1bd-085897d9a635","Type":"ContainerStarted","Data":"7ae340e656a6162e5721f1e18d4e4f2a98ca01b8e223b059070a329b7cb62227"}
Nov 03 20:07:15.762706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.513046    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:15.828876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.825294    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.829454 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:15.829411    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c616662_7a72_4200_b53e_533f3b6a21f0.slice/crio-983228d8e308d373e22092e1a03c974f8119e35ca945646d3f9d39da8fa124bc\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c616662_7a72_4200_b53e_533f3b6a21f0.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:07:15.842460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.842435    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"]
Nov 03 20:07:15.842676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.842662    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1cca2dba-545c59947d-wqr4r"]
Nov 03 20:07:15.842853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.842827    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:07:15.842976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.842966    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"]
Nov 03 20:07:15.843105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.843095    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:07:15.843286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.843276    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:07:15.843500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.843490    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:07:15.843634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.843624    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:07:15.843762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.843752    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:07:15.843873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.843864    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1cca2dba-77444d6cb7-47x79"]
Nov 03 20:07:15.855751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.855645    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:15.870225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.867056    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"]
Nov 03 20:07:15.899148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.898176    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:15.912760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:15.912674    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.186477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.167807    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"]
Nov 03 20:07:16.440277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.402073    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.462142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.462079    5282 cpu_manager.go:414] "containerMap: removing container" podUID="02059338-a95a-4f71-9e66-8d9b4e6e6646" containerName="payload-pre-processing"
Nov 03 20:07:16.462142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.462112    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02059338-a95a-4f71-9e66-8d9b4e6e6646" containerName="payload-pre-processing"
Nov 03 20:07:16.462142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.462134    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3c616662-7a72-4200-b53e-533f3b6a21f0" containerName="payload-processing"
Nov 03 20:07:16.462142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.462142    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c616662-7a72-4200-b53e-533f3b6a21f0" containerName="payload-processing"
Nov 03 20:07:16.470638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.462330    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="02059338-a95a-4f71-9e66-8d9b4e6e6646" containerName="payload-pre-processing"
Nov 03 20:07:16.470638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.462344    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c616662-7a72-4200-b53e-533f3b6a21f0" containerName="payload-processing"
Nov 03 20:07:16.478718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.478690    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"]
Nov 03 20:07:16.486771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.486663    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.520233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.511706    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d81d18-openshift-default-dockercfg-7vnl7\""
Nov 03 20:07:16.578661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.578523    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.578661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.578591    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgthg\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-kube-api-access-zgthg\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.578661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.578641    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbfc761-e296-4114-b4e3-da2df1301934-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.578965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.578703    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.578965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.578879    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.579086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.578965    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.579086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.579023    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbfc761-e296-4114-b4e3-da2df1301934-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.579086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.579068    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.579271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.579111    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732104    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732214    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732276    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbfc761-e296-4114-b4e3-da2df1301934-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732322    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732362    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732443    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732479    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgthg\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-kube-api-access-zgthg\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732520    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbfc761-e296-4114-b4e3-da2df1301934-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.732579    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.733337    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.765248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.751646    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.810948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.791777    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.810948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.793317    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.813715 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.813681    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:16.829132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.829070    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgthg\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-kube-api-access-zgthg\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.858220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.855773    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:07:16.900162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.899618    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbfc761-e296-4114-b4e3-da2df1301934-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.900162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.899841    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:16.929437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.929364    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.978886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.952958    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm" event={"ID":"3f3620eb-8848-4b9b-a1bd-085897d9a635","Type":"ContainerStarted","Data":"149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1"}
Nov 03 20:07:16.978886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.959557    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:16.978886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:16.978114    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbfc761-e296-4114-b4e3-da2df1301934-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:17.036752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.036679    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm" podStartSLOduration=8.036646735 podStartE2EDuration="8.036646735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:16.993124536 +0000 UTC m=+1629.662326007" watchObservedRunningTime="2026-11-03 20:07:17.036646735 +0000 UTC m=+1629.705848205"
Nov 03 20:07:17.043744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.043714    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:17.052045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.050481    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:07:17.059716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.057511    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"]
Nov 03 20:07:17.059716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.057768    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" podUID="5710754b-9d46-4eb8-a3bf-032b7268e2e0" containerName="maas-api" containerID="cri-o://8c972fcb4fa9de6ac1d2afeb24cd7299a3abe27ce2259b46b7f651554cdceb9a" gracePeriod=30
Nov 03 20:07:17.068556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.067294    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.123225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.121150    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:07:17.212239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.212086    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.359737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.359612    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.792056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.790384    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:17.792056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.790495    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:17.792056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.790546    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:17.954261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.953788    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02059338-a95a-4f71-9e66-8d9b4e6e6646" path="/var/lib/kubelet/pods/02059338-a95a-4f71-9e66-8d9b4e6e6646/volumes"
Nov 03 20:07:17.963135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.957185    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:17.963135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.957648    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c616662-7a72-4200-b53e-533f3b6a21f0" path="/var/lib/kubelet/pods/3c616662-7a72-4200-b53e-533f3b6a21f0/volumes"
Nov 03 20:07:17.963135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.958064    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"]
Nov 03 20:07:17.963135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.958119    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:17.963135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.958192    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:17.971377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.970377    5282 generic.go:356] "Generic (PLEG): container finished" podID="5710754b-9d46-4eb8-a3bf-032b7268e2e0" containerID="8c972fcb4fa9de6ac1d2afeb24cd7299a3abe27ce2259b46b7f651554cdceb9a" exitCode=0
Nov 03 20:07:17.971377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.970740    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" event={"ID":"5710754b-9d46-4eb8-a3bf-032b7268e2e0","Type":"ContainerDied","Data":"8c972fcb4fa9de6ac1d2afeb24cd7299a3abe27ce2259b46b7f651554cdceb9a"}
Nov 03 20:07:17.971377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.970892    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh" event={"ID":"5710754b-9d46-4eb8-a3bf-032b7268e2e0","Type":"ContainerDied","Data":"cbf3507ec71fd9d85c12d69fd98cbdf3e4eaaa23b35720af2262a1e26514649e"}
Nov 03 20:07:17.971377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:17.970923    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf3507ec71fd9d85c12d69fd98cbdf3e4eaaa23b35720af2262a1e26514649e"
Nov 03 20:07:18.010013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.008954    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" event={"ID":"dfbfc761-e296-4114-b4e3-da2df1301934","Type":"ContainerStarted","Data":"1f0d4133a98caf6d6b091998f7bf25b640d31fcfcf9e62749073e971f87dc41c"}
Nov 03 20:07:18.010013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.010004    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:07:18.010317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.010298    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:18.014536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.014522    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:07:18.126019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.125940    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:18.155772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.155735    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\") pod \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") "
Nov 03 20:07:18.158845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.158813    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5710754b-9d46-4eb8-a3bf-032b7268e2e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5710754b-9d46-4eb8-a3bf-032b7268e2e0-openshift-service-ca\") pod \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") "
Nov 03 20:07:18.165457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.163682    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5710754b-9d46-4eb8-a3bf-032b7268e2e0-kube-api-access-tqk6n\" (UniqueName: \"kubernetes.io/projected/5710754b-9d46-4eb8-a3bf-032b7268e2e0-kube-api-access-tqk6n\") pod \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") "
Nov 03 20:07:18.165457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.164318    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-metrics-certs\") pod \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\" (UID: \"5710754b-9d46-4eb8-a3bf-032b7268e2e0\") "
Nov 03 20:07:18.166261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.163402    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5710754b-9d46-4eb8-a3bf-032b7268e2e0-openshift-service-ca" pod "5710754b-9d46-4eb8-a3bf-032b7268e2e0" (UID: "5710754b-9d46-4eb8-a3bf-032b7268e2e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:18.169324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.169281    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5710754b-9d46-4eb8-a3bf-032b7268e2e0-kube-api-access-tqk6n" pod "5710754b-9d46-4eb8-a3bf-032b7268e2e0" (UID: "5710754b-9d46-4eb8-a3bf-032b7268e2e0"). InnerVolumeSpecName "kube-api-access-tqk6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:18.169530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.169495    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls" pod "5710754b-9d46-4eb8-a3bf-032b7268e2e0" (UID: "5710754b-9d46-4eb8-a3bf-032b7268e2e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:18.171707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.171680    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-metrics-certs" pod "5710754b-9d46-4eb8-a3bf-032b7268e2e0" (UID: "5710754b-9d46-4eb8-a3bf-032b7268e2e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:07:18.263313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.263251    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:18.268682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.268654    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5710754b-9d46-4eb8-a3bf-032b7268e2e0-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:18.268682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.268681    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqk6n\" (UniqueName: \"kubernetes.io/projected/5710754b-9d46-4eb8-a3bf-032b7268e2e0-kube-api-access-tqk6n\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:18.268844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.268694    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:18.268844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.268704    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5710754b-9d46-4eb8-a3bf-032b7268e2e0-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:18.883221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.872396    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"]
Nov 03 20:07:18.883221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.872636    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq" podUID="d1255c5a-19f1-44e2-a8a7-70d0da40b018" containerName="istio-proxy" containerID="cri-o://3a68469c1c5b05783badae65fbcc53b9f20b8a7eaba305cc21937a52b5c09855" gracePeriod=30
Nov 03 20:07:18.937714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:18.935096    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.199315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.191091    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.402473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.402423    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.515220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.513392    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"
Nov 03 20:07:19.517135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.517110    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.526796 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.526759    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" event={"ID":"dfbfc761-e296-4114-b4e3-da2df1301934","Type":"ContainerStarted","Data":"1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5"}
Nov 03 20:07:19.553252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.548629    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podStartSLOduration=3.548602219 podStartE2EDuration="3.548602219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:19.545825566 +0000 UTC m=+1632.215027036" watchObservedRunningTime="2026-11-03 20:07:19.548602219 +0000 UTC m=+1632.217803680"
Nov 03 20:07:19.564913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.564882    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"]
Nov 03 20:07:19.570528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.570503    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1cca2dba-857d57dff5-lhgwh"]
Nov 03 20:07:19.654663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.654624    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.949056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.949001    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:19.949878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.949860    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"]
Nov 03 20:07:19.951365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.951345    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5710754b-9d46-4eb8-a3bf-032b7268e2e0" containerName="maas-api"
Nov 03 20:07:19.951488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.951476    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5710754b-9d46-4eb8-a3bf-032b7268e2e0" containerName="maas-api"
Nov 03 20:07:19.951767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.951753    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5710754b-9d46-4eb8-a3bf-032b7268e2e0" containerName="maas-api"
Nov 03 20:07:19.957974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.957944    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"]
Nov 03 20:07:19.958091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.958071    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:19.968901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:19.968860    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-323cb6ed-openshift-default-dockercfg-nqstq\""
Nov 03 20:07:20.044860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.044765    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:20.052678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.051671    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:20.055027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.054881    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-token\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.054983    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-credential-socket\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.055085    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-podinfo\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.055160    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-data\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.055305    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-envoy\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.055436    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-socket\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.055555    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lct\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-kube-api-access-f7lct\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.055593    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-certs\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.055780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.055634    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istiod-ca-cert\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.077905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.077880    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:20.139083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.139049    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"]
Nov 03 20:07:20.143702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.143679    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:20.153563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.153538    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c13f7966\""
Nov 03 20:07:20.155359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.155334    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"]
Nov 03 20:07:20.157316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157292    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-envoy\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157378    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczln\" (UniqueName: \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\") pod \"payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:20.157452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157430    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-socket\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157480    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lct\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-kube-api-access-f7lct\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157514    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-certs\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157555    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istiod-ca-cert\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157606    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:20.157757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157678    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-token\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157771    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-credential-socket\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157846    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-podinfo\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.157952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.157905    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-data\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.158348 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.158327    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-data\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.158485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.158368    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-certs\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.159272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.159247    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-credential-socket\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.159406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.159385    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istiod-ca-cert\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.159903 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.159879    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-socket\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.172504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.172478    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-podinfo\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.176051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.176026    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-envoy\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.176458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.176437    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c13f7966-dockercfg-lcqs2\""
Nov 03 20:07:20.186475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.186441    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"]
Nov 03 20:07:20.193066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.193046    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.196979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.196955    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-token\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.198264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.198243    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lct\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-kube-api-access-f7lct\") pod \"e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") " pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.201459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.201436    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"]
Nov 03 20:07:20.221746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.221720    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:20.318278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.259980    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczln\" (UniqueName: \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\") pod \"payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:20.318278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.260108    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:20.318278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.260154    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktfc\" (UniqueName: \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\") pod \"payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") " pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.318278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.260286    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") " pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.318278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.290544    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:20.456410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.436006    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:20.475410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.475376    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktfc\" (UniqueName: \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\") pod \"payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") " pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.487332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.487275    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") " pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.605526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.604375    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") " pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.615098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.615048    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktfc\" (UniqueName: \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\") pod \"payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") " pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.616814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.616774    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:20.625232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.625210    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:20.638310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.638281    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:07:20.724849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.724814    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczln\" (UniqueName: \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\") pod \"payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:20.754406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.754375    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:20.765930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:20.765705    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:21.052979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.052828    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"]
Nov 03 20:07:21.056840 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:21.056808    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c2d5d9_4c87_4e8c_8013_8518870b1ed8.slice/crio-d96cf38c59315067309aea49d6594e0b2c0649bbd4f41f46519b91663737a0b5 WatchSource:0}: Error finding container d96cf38c59315067309aea49d6594e0b2c0649bbd4f41f46519b91663737a0b5: Status 404 returned error can't find the container with id d96cf38c59315067309aea49d6594e0b2c0649bbd4f41f46519b91663737a0b5
Nov 03 20:07:21.065666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.065623    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:21.065814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.065735    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:21.065937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.065917    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:07:21.095102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.095076    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"]
Nov 03 20:07:21.095727 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:21.095699    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393a5585_80b4_4d74_ab5e_51eab898498f.slice/crio-906446e11af11793bf7ecf9534e14a5c59c1503684a62db197304888ef9ea2e4 WatchSource:0}: Error finding container 906446e11af11793bf7ecf9534e14a5c59c1503684a62db197304888ef9ea2e4: Status 404 returned error can't find the container with id 906446e11af11793bf7ecf9534e14a5c59c1503684a62db197304888ef9ea2e4
Nov 03 20:07:21.109375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.109354    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"]
Nov 03 20:07:21.110727 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:21.110699    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead6acea_c6d3_4e38_bedb_dcb9541bcd3c.slice/crio-5de264d60c7e7dd361c21fef3966767fa17b94e4c41b1c59cc9e6e3e9b12ca9a WatchSource:0}: Error finding container 5de264d60c7e7dd361c21fef3966767fa17b94e4c41b1c59cc9e6e3e9b12ca9a: Status 404 returned error can't find the container with id 5de264d60c7e7dd361c21fef3966767fa17b94e4c41b1c59cc9e6e3e9b12ca9a
Nov 03 20:07:21.323144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.323108    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:21.458697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.458618    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5710754b-9d46-4eb8-a3bf-032b7268e2e0" path="/var/lib/kubelet/pods/5710754b-9d46-4eb8-a3bf-032b7268e2e0/volumes"
Nov 03 20:07:21.683223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.682950    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx" event={"ID":"12c2d5d9-4c87-4e8c-8013-8518870b1ed8","Type":"ContainerStarted","Data":"0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662"}
Nov 03 20:07:21.683223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.683017    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx" event={"ID":"12c2d5d9-4c87-4e8c-8013-8518870b1ed8","Type":"ContainerStarted","Data":"d96cf38c59315067309aea49d6594e0b2c0649bbd4f41f46519b91663737a0b5"}
Nov 03 20:07:21.696851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.696801    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2" event={"ID":"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c","Type":"ContainerStarted","Data":"a1f6f7f8fa3ccfefc169bd5866b87e181e81bc9f007e4978e464bae903a69482"}
Nov 03 20:07:21.697133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.697103    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2" event={"ID":"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c","Type":"ContainerStarted","Data":"5de264d60c7e7dd361c21fef3966767fa17b94e4c41b1c59cc9e6e3e9b12ca9a"}
Nov 03 20:07:21.707491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.703384    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:21.707491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.706833    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx" event={"ID":"393a5585-80b4-4d74-ab5e-51eab898498f","Type":"ContainerStarted","Data":"9be12bf7b96c80d5af59771f6862bf6360914475f9504dea973e8c53b29b51ed"}
Nov 03 20:07:21.707491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.706887    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx" event={"ID":"393a5585-80b4-4d74-ab5e-51eab898498f","Type":"ContainerStarted","Data":"906446e11af11793bf7ecf9534e14a5c59c1503684a62db197304888ef9ea2e4"}
Nov 03 20:07:21.707491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.707382    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:21.716998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.716896    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx" podStartSLOduration=2.716878809 podStartE2EDuration="2.716878809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:21.712023494 +0000 UTC m=+1634.381225001" watchObservedRunningTime="2026-11-03 20:07:21.716878809 +0000 UTC m=+1634.386080280"
Nov 03 20:07:21.729305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.729248    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx" podStartSLOduration=1.729231242 podStartE2EDuration="1.729231242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:21.728016987 +0000 UTC m=+1634.397218457" watchObservedRunningTime="2026-11-03 20:07:21.729231242 +0000 UTC m=+1634.398432712"
Nov 03 20:07:21.748482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:21.748432    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2" podStartSLOduration=1.748420833 podStartE2EDuration="1.748420833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:21.743901669 +0000 UTC m=+1634.413103139" watchObservedRunningTime="2026-11-03 20:07:21.748420833 +0000 UTC m=+1634.417622303"
Nov 03 20:07:22.115530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.115497    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.269272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.269230    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.308257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.308220    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:22.315168 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.315136    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:22.714393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.701788    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:22.789701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.789645    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:22.794476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.794442    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:07:22.823272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.822296    5282 patch_prober.go:29] interesting pod/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:07:22.823272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:22.822379    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq" podUID="d1255c5a-19f1-44e2-a8a7-70d0da40b018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:23.393962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.393930    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:23.405449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.405416    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"]
Nov 03 20:07:23.410937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.410916    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.415946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.415920    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-d81d18\""
Nov 03 20:07:23.418836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.418812    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"]
Nov 03 20:07:23.429703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.429677    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"]
Nov 03 20:07:23.435021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.434999    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.443809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.443778    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"]
Nov 03 20:07:23.448712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.446864    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-d81d18-dockercfg-rq75n\""
Nov 03 20:07:23.555222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.543819    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.555222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.544048    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7blkn\" (UniqueName: \"kubernetes.io/projected/03dece0c-cf3c-4d61-833d-a7a8a3531666-kube-api-access-7blkn\") pod \"payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.555222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.544143    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5w9\" (UniqueName: \"kubernetes.io/projected/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-kube-api-access-sc5w9\") pod \"payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.555222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.544180    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03dece0c-cf3c-4d61-833d-a7a8a3531666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.555570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.555245    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:23.652280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.651781    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7blkn\" (UniqueName: \"kubernetes.io/projected/03dece0c-cf3c-4d61-833d-a7a8a3531666-kube-api-access-7blkn\") pod \"payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.652280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.651882    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5w9\" (UniqueName: \"kubernetes.io/projected/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-kube-api-access-sc5w9\") pod \"payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.652280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.651929    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03dece0c-cf3c-4d61-833d-a7a8a3531666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.652280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.652121    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.672136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.653159    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.672136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.671264    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03dece0c-cf3c-4d61-833d-a7a8a3531666-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.739905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.709933    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5w9\" (UniqueName: \"kubernetes.io/projected/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-kube-api-access-sc5w9\") pod \"payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.739905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.738636    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:23.742845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.742815    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7blkn\" (UniqueName: \"kubernetes.io/projected/03dece0c-cf3c-4d61-833d-a7a8a3531666-kube-api-access-7blkn\") pod \"payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.786984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.785008    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:23.867877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.867670    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:23.997833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:23.990759    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:24.285093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.266807    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:24.326584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.320256    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"]
Nov 03 20:07:24.326584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.320342    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"]
Nov 03 20:07:24.364726 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:24.364694    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd7dabe_9b92_4f7d_be7a_0f9e3b4d22c3.slice/crio-3c265724397d866ed10ce13f7fba972b09ac0a167b4d965b5e2835747276d202 WatchSource:0}: Error finding container 3c265724397d866ed10ce13f7fba972b09ac0a167b4d965b5e2835747276d202: Status 404 returned error can't find the container with id 3c265724397d866ed10ce13f7fba972b09ac0a167b4d965b5e2835747276d202
Nov 03 20:07:24.857279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.857238    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq" event={"ID":"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3","Type":"ContainerStarted","Data":"afb46bcc5ffda08cd1ac2fca2c0e89278a02009c07fc2956d059cc312ceee774"}
Nov 03 20:07:24.857722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.857334    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:24.857722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.857365    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq" event={"ID":"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3","Type":"ContainerStarted","Data":"3c265724397d866ed10ce13f7fba972b09ac0a167b4d965b5e2835747276d202"}
Nov 03 20:07:24.859110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.859077    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh" event={"ID":"03dece0c-cf3c-4d61-833d-a7a8a3531666","Type":"ContainerStarted","Data":"b29d786134db65e08463275384b6591b4eabc12250eb6310c5d4a5e01b0c4aa9"}
Nov 03 20:07:24.859246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.859133    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh" event={"ID":"03dece0c-cf3c-4d61-833d-a7a8a3531666","Type":"ContainerStarted","Data":"346d031549f8748a7c7d9aeb72f65ecd42a740aa3c021d00c89cce91477aeea0"}
Nov 03 20:07:24.859246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.859161    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:24.880653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.880601    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq" podStartSLOduration=1.8805863779999998 podStartE2EDuration="1.880586378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:24.879919031 +0000 UTC m=+1637.549120493" watchObservedRunningTime="2026-11-03 20:07:24.880586378 +0000 UTC m=+1637.549787848"
Nov 03 20:07:24.898258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.898161    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh" podStartSLOduration=1.8981479239999999 podStartE2EDuration="1.898147924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:24.896679237 +0000 UTC m=+1637.565880708" watchObservedRunningTime="2026-11-03 20:07:24.898147924 +0000 UTC m=+1637.567349454"
Nov 03 20:07:24.903786 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:24.903757    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.199329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.198027    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.240486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.240456    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"]
Nov 03 20:07:25.248537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.248503    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:25.252382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.252353    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-323cb6ed-dockercfg-5ms86\""
Nov 03 20:07:25.252522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.252395    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-323cb6ed\""
Nov 03 20:07:25.252974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.252955    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"]
Nov 03 20:07:25.348634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.337846    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdz29\" (UniqueName: \"kubernetes.io/projected/326e6d8b-4ca9-4481-95e1-25aca464e712-kube-api-access-vdz29\") pod \"payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:25.348634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.337971    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326e6d8b-4ca9-4481-95e1-25aca464e712-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:25.384815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.361441    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:25.455277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.450150    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdz29\" (UniqueName: \"kubernetes.io/projected/326e6d8b-4ca9-4481-95e1-25aca464e712-kube-api-access-vdz29\") pod \"payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:25.455277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.450285    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326e6d8b-4ca9-4481-95e1-25aca464e712-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:25.553266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.538082    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdz29\" (UniqueName: \"kubernetes.io/projected/326e6d8b-4ca9-4481-95e1-25aca464e712-kube-api-access-vdz29\") pod \"payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:25.639239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.629480    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326e6d8b-4ca9-4481-95e1-25aca464e712-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:25.867255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.867224    5282 generic.go:356] "Generic (PLEG): container finished" podID="d1255c5a-19f1-44e2-a8a7-70d0da40b018" containerID="3a68469c1c5b05783badae65fbcc53b9f20b8a7eaba305cc21937a52b5c09855" exitCode=0
Nov 03 20:07:25.867255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.867245    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq" event={"ID":"d1255c5a-19f1-44e2-a8a7-70d0da40b018","Type":"ContainerDied","Data":"3a68469c1c5b05783badae65fbcc53b9f20b8a7eaba305cc21937a52b5c09855"}
Nov 03 20:07:25.871032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:25.871011    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:26.046099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.046057    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:07:26.049014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.048988    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:26.256083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.255703    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"]
Nov 03 20:07:26.259283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259260    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-socket\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.259579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259372    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-credential-socket\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.259579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259419    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-kube-api-access-zpqn2\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-kube-api-access-zpqn2\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.259776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259577    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-podinfo\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.259776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259632    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-token\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-token\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.259776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259646    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-credential-socket" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:26.259776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259741    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-data\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.259981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259859    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-envoy\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.259981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259939    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istiod-ca-cert\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.260084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.259983    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-certs\") pod \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\" (UID: \"d1255c5a-19f1-44e2-a8a7-70d0da40b018\") "
Nov 03 20:07:26.261012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.260986    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-certs" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:26.261192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.261171    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-data" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:26.261633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.261611    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.261938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.261916    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-socket" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:26.262623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.262597    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istiod-ca-cert" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:26.263791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.263723    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-token" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:26.263939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.263918    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-envoy" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:07:26.264448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.264386    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-podinfo" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:07:26.264522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.264473    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-kube-api-access-zpqn2" pod "d1255c5a-19f1-44e2-a8a7-70d0da40b018" (UID: "d1255c5a-19f1-44e2-a8a7-70d0da40b018"). InnerVolumeSpecName "kube-api-access-zpqn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:26.362982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.362954    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqn2\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-kube-api-access-zpqn2\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.362982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.362981    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.363332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.362993    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.363332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.363002    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.363332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.363011    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.363332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.363020    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1255c5a-19f1-44e2-a8a7-70d0da40b018-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.363332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.363029    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.363332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.363037    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1255c5a-19f1-44e2-a8a7-70d0da40b018-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:26.875634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.875600    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm" event={"ID":"326e6d8b-4ca9-4481-95e1-25aca464e712","Type":"ContainerStarted","Data":"da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735"}
Nov 03 20:07:26.876129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.875654    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm" event={"ID":"326e6d8b-4ca9-4481-95e1-25aca464e712","Type":"ContainerStarted","Data":"55a59bd43e7d2852f7bd2715e900ceca82a81f212e11884c67a3a61b4d88dbea"}
Nov 03 20:07:26.876129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.875676    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:26.877069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.877047    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"
Nov 03 20:07:26.877174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.877080    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq" event={"ID":"d1255c5a-19f1-44e2-a8a7-70d0da40b018","Type":"ContainerDied","Data":"fd5e93c5065be3b4efc13dcff7aa05e207b29c54fa42fcfd5b54be795daf5ab7"}
Nov 03 20:07:26.877174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.877129    5282 scope.go:122] "RemoveContainer" containerID="3a68469c1c5b05783badae65fbcc53b9f20b8a7eaba305cc21937a52b5c09855"
Nov 03 20:07:26.892414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.892372    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm" podStartSLOduration=1.8923579560000001 podStartE2EDuration="1.892357956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:26.890176121 +0000 UTC m=+1639.559377592" watchObservedRunningTime="2026-11-03 20:07:26.892357956 +0000 UTC m=+1639.561559425"
Nov 03 20:07:26.892532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.892472    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:26.906571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.906544    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"]
Nov 03 20:07:26.909918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:26.909898    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ea700cd3-openshift-default-5557655bb7-pdbpq"]
Nov 03 20:07:27.488783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:27.488751    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1255c5a-19f1-44e2-a8a7-70d0da40b018" path="/var/lib/kubelet/pods/d1255c5a-19f1-44e2-a8a7-70d0da40b018/volumes"
Nov 03 20:07:27.616971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:27.616933    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"]
Nov 03 20:07:27.618731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:27.618702    5282 cpu_manager.go:414] "containerMap: removing container" podUID="d1255c5a-19f1-44e2-a8a7-70d0da40b018" containerName="istio-proxy"
Nov 03 20:07:27.618912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:27.618895    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1255c5a-19f1-44e2-a8a7-70d0da40b018" containerName="istio-proxy"
Nov 03 20:07:27.632222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:27.619274    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1255c5a-19f1-44e2-a8a7-70d0da40b018" containerName="istio-proxy"
Nov 03 20:07:27.711571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:27.708018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:27.999220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:27.999171    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.000085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.000033    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"]
Nov 03 20:07:28.006521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.006384    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c13f7966-serving-cert\""
Nov 03 20:07:28.025366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.007154    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:28.121275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.101393    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-metrics-certs\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.121275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.101535    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpllj\" (UniqueName: \"kubernetes.io/projected/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-kube-api-access-bpllj\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.121275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.101673    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-maas-api-tls\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.121275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.101912    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-openshift-service-ca\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.204338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.203627    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-openshift-service-ca\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.204338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.203756    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-metrics-certs\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.204338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.203819    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpllj\" (UniqueName: \"kubernetes.io/projected/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-kube-api-access-bpllj\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.204338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.203895    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-maas-api-tls\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.205330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.205296    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-openshift-service-ca\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.208014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.206980    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-maas-api-tls\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.210978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.209676    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-metrics-certs\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.216137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.216112    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpllj\" (UniqueName: \"kubernetes.io/projected/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-kube-api-access-bpllj\") pod \"maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.356643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.356562    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:28.780421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:28.780387    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:29.092231 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:29.090307    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5923c2fe_8bb2_43c4_b1a6_c28f0e9bc970.slice/crio-c39a522b7760fc5cbfb881a6b7bc75996563f2ec5b8d615ffcecd66a84dd8649 WatchSource:0}: Error finding container c39a522b7760fc5cbfb881a6b7bc75996563f2ec5b8d615ffcecd66a84dd8649: Status 404 returned error can't find the container with id c39a522b7760fc5cbfb881a6b7bc75996563f2ec5b8d615ffcecd66a84dd8649
Nov 03 20:07:29.111314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:29.108414    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"]
Nov 03 20:07:29.596737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:29.596710    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:30.124916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:30.124882    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz" event={"ID":"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970","Type":"ContainerStarted","Data":"832c02ee953995fae6104d8a9cdcea69c279b98c611024f0750f820254c99747"}
Nov 03 20:07:30.125408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:30.124932    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz" event={"ID":"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970","Type":"ContainerStarted","Data":"c39a522b7760fc5cbfb881a6b7bc75996563f2ec5b8d615ffcecd66a84dd8649"}
Nov 03 20:07:30.125408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:30.125092    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:30.144616 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:30.144562    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz" podStartSLOduration=2.688998499 podStartE2EDuration="3.144548126s" totalImagesPullingTime="455.549627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:30.141631737 +0000 UTC m=+1642.810833207" watchObservedRunningTime="2026-11-03 20:07:30.144548126 +0000 UTC m=+1642.813749629"
Nov 03 20:07:30.430238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:30.430117    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:30.593222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:30.586323    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.308987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:31.308949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:31.742821 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:31.742785    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:07:32.004554 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.004471    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"]
Nov 03 20:07:32.009324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.009294    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.015400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.015376    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-323cb6ed-serving-cert\""
Nov 03 20:07:32.022100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.022067    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"]
Nov 03 20:07:32.061738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.061486    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-metrics-certs\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.061912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.061846    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.061912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.061888    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562bp\" (UniqueName: \"kubernetes.io/projected/f327a1f3-97da-4c0d-820d-3bd29cee71ca-kube-api-access-562bp\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.062163 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.062137    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f327a1f3-97da-4c0d-820d-3bd29cee71ca-openshift-service-ca\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.091793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.091765    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.208233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.186879    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.208233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.186953    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562bp\" (UniqueName: \"kubernetes.io/projected/f327a1f3-97da-4c0d-820d-3bd29cee71ca-kube-api-access-562bp\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.208233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.187226    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f327a1f3-97da-4c0d-820d-3bd29cee71ca-openshift-service-ca\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.208233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.187483    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-metrics-certs\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.208233 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:32.191738    5282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-serving-cert: secret "maas-api-e2e-ait-323cb6ed-serving-cert" not found
Nov 03 20:07:32.208233 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:32.191817    5282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls podName:f327a1f3-97da-4c0d-820d-3bd29cee71ca nodeName:}" failed. No retries permitted until 2026-11-03 20:07:32.691795711 +0000 UTC m=+1645.360997173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls") pod "maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql" (UID: "f327a1f3-97da-4c0d-820d-3bd29cee71ca") : secret "maas-api-e2e-ait-323cb6ed-serving-cert" not found
Nov 03 20:07:32.208233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.193247    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f327a1f3-97da-4c0d-820d-3bd29cee71ca-openshift-service-ca\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.233746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.225024    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-metrics-certs\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.242485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.241634    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562bp\" (UniqueName: \"kubernetes.io/projected/f327a1f3-97da-4c0d-820d-3bd29cee71ca-kube-api-access-562bp\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.284325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.256799    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.405637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.405585    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:32.725298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.714416    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.732225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.729922    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls\") pod \"maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:32.933232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:32.932499    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:33.051756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:33.051714    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:33.490985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:33.490958    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"]
Nov 03 20:07:33.491777 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:33.491754    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf327a1f3_97da_4c0d_820d_3bd29cee71ca.slice/crio-e400b9c2399ce71d11ee930f7d3cce950c1dfbbed7ab3d028eef064c9c1fd6af WatchSource:0}: Error finding container e400b9c2399ce71d11ee930f7d3cce950c1dfbbed7ab3d028eef064c9c1fd6af: Status 404 returned error can't find the container with id e400b9c2399ce71d11ee930f7d3cce950c1dfbbed7ab3d028eef064c9c1fd6af
Nov 03 20:07:33.760606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:33.760518    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:33.863135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:33.861521    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:33.863135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:33.862846    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:34.312771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:34.312736    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql" event={"ID":"f327a1f3-97da-4c0d-820d-3bd29cee71ca","Type":"ContainerStarted","Data":"3d6497c18a03b9c133b881a480397b9cfdd9460b6e80d2eb85e454880e0c00a8"}
Nov 03 20:07:34.312973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:34.312795    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:34.312973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:34.312812    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql" event={"ID":"f327a1f3-97da-4c0d-820d-3bd29cee71ca","Type":"ContainerStarted","Data":"e400b9c2399ce71d11ee930f7d3cce950c1dfbbed7ab3d028eef064c9c1fd6af"}
Nov 03 20:07:34.349633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:34.349574    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql" podStartSLOduration=2.9226201830000003 podStartE2EDuration="3.349560295s" totalImagesPullingTime="426.940112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:34.346021628 +0000 UTC m=+1647.015223099" watchObservedRunningTime="2026-11-03 20:07:34.349560295 +0000 UTC m=+1647.018761764"
Nov 03 20:07:34.512258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:34.511821    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:35.266220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:35.262749    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:35.870133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:35.870103    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:07:35.870706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:35.870348    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:07:35.902068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:35.902013    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:36.187916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:36.187838    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:07:36.696049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:36.696009    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.409410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:37.409373    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:37.997735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:37.997707    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:07:38.220369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:38.220324    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:38.936865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:38.936834    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:40.090712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:40.090682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:40.372788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:40.372712    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:07:40.883261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:40.883193    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:41.698265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:41.691541    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:41.931457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:41.931416    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.353467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:42.353431    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.485564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:42.485533    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.807512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:42.807482    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:42.947220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:42.943424    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:43.534393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:43.532661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:44.467084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:44.467049    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:45.292002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:45.291972    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.196835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:46.196796    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.716670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:46.716635    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:46.848978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:46.848855    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:47.135830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:47.135800    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:47.213581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:47.213549    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:07:47.984635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:47.984599    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.080681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.080647    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"]
Nov 03 20:07:48.095895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.095856    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"]
Nov 03 20:07:48.096072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.095965    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.115968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.115927    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.162277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.148217    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-944e83-dockercfg-ds6nr\""
Nov 03 20:07:48.162277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.148393    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-944e83\""
Nov 03 20:07:48.236754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.236607    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ktw\" (UniqueName: \"kubernetes.io/projected/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-kube-api-access-22ktw\") pod \"payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.236754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.236683    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.346223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.345355    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ktw\" (UniqueName: \"kubernetes.io/projected/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-kube-api-access-22ktw\") pod \"payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.346223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.345437    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.361221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.346679    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.376268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.371282    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:48.404222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.397231    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ktw\" (UniqueName: \"kubernetes.io/projected/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-kube-api-access-22ktw\") pod \"payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.425221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.417924    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:48.854276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:48.842981    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"]
Nov 03 20:07:49.253253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:49.253220    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"]
Nov 03 20:07:49.387356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:49.385777    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:49.387356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:49.385984    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"]
Nov 03 20:07:49.560231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:49.546365    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:50.227018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.226967    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 03 20:07:50.473240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.472366    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:50.590040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.588412    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:50.590040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.589404    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:50.602667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.602621    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:50.634040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.633994    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:07:50.641395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.641353    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:50.673562 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.673516    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:50.677330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.677292    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:50.693214 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:50.693173    5282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.229s"
Nov 03 20:07:50.693401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.693247    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph" event={"ID":"0ecdc4e5-5885-4c49-85b4-a58ed83fd584","Type":"ContainerStarted","Data":"a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d"}
Nov 03 20:07:50.693401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.693377    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph" event={"ID":"0ecdc4e5-5885-4c49-85b4-a58ed83fd584","Type":"ContainerStarted","Data":"a68b8135d5ea6eac9a74180db074460dc16dc9645dcb015a2262ac06bd240367"}
Nov 03 20:07:50.693533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.693409    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:07:50.717276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.711132    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph" podStartSLOduration=2.711109924 podStartE2EDuration="2.711109924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:50.67488388 +0000 UTC m=+1663.344085352" watchObservedRunningTime="2026-11-03 20:07:50.711109924 +0000 UTC m=+1663.380311395"
Nov 03 20:07:50.999336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:50.999292    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.001185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.001151    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.054469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.054423    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.055405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.054717    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.452400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.452293    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.537831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.537794    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:51.592152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.592100    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.594006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.593542    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.595534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.594750    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:07:51.596379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.596361    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:07:51.601364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.601328    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.606300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.606260    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.608485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.608468    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:07:51.642258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.640842    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.643399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.643377    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:07:51.655223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.644603    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.655223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.645658    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:07:51.677831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.674652    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:51.677831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.675269    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:07:51.694394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.694341    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:51.695956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.695926    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:07:51.999911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:51.999856    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:52.000699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.000677    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:07:52.062552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.062504    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:52.063094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.063060    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:07:52.083459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.080531    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:07:52.103806 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.103760    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:07:52.552877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.552324    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:52.591283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.591245    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 20:07:52.732254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.607876    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:52.734443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.607918    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:52.754223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.607952    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:52.754223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.746334    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:07:52.754223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.608540    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:52.754223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.656371    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:52.754223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.680353    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:52.758937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.658382    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:52.759261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.699082    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:52.791798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:52.687717    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:53.032287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.005286    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:53.032287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.006847    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:53.138224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.116219    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:07:53.249223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.117161    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:07:53.313366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.294175    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:07:53.333958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.333923    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:07:53.676285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.674610    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:53.797240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.783653    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:53.797240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.793973    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 03 20:07:53.797240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.796723    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:53.822765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.822717    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:53.843228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.824323    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:53.913225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.896647    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:53.970117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.948850    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:07:53.970117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.948939    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.180629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.978222    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:07:54.180629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.978304    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.180629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:53.978492    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.180629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.019456    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:07:54.180629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.055461    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.180629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.127286    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.380228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.351264    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:54.380228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.359366    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx" podUID="393a5585-80b4-4d74-ab5e-51eab898498f" containerName="payload-processing" containerID="cri-o://9be12bf7b96c80d5af59771f6862bf6360914475f9504dea973e8c53b29b51ed" gracePeriod=25
Nov 03 20:07:54.380228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.368442    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.392039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.391983    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:54.453404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.447408    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2" podUID="ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" containerName="payload-pre-processing" containerID="cri-o://a1f6f7f8fa3ccfefc169bd5866b87e181e81bc9f007e4978e464bae903a69482" gracePeriod=25
Nov 03 20:07:54.559192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.559151    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:07:54.588324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.572002    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:07:54.609442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.609406    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:54.815451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.801790    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.815451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.801885    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:07:54.921222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.908572    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.921222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.908926    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:54.921222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.909013    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:54.985226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.984624    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:54.990912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:54.989533    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.027884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.024224    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.059391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.059286    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.166251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.136677    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.252277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.167695    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"]
Nov 03 20:07:55.252277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.216265    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:55.384928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.384830    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.385636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.385612    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:07:55.388155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.388120    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.437318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.427704    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:55.787223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.757763    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:07:55.840368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.794805    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:55.916815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.912633    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.940223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.922680    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.940223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.933043    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.940223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.913876    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:55.967221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.949282    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.094979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:55.997362    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.094979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.001365    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:07:56.094979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.011554    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:07:56.160335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.152029    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.183946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.171549    5282 generic.go:356] "Generic (PLEG): container finished" podID="ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" containerID="a1f6f7f8fa3ccfefc169bd5866b87e181e81bc9f007e4978e464bae903a69482" exitCode=0
Nov 03 20:07:56.183946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.171674    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2" event={"ID":"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c","Type":"ContainerDied","Data":"a1f6f7f8fa3ccfefc169bd5866b87e181e81bc9f007e4978e464bae903a69482"}
Nov 03 20:07:56.210652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.191358    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.308291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.301988    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.423440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.388590    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.456440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.388637    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.534221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.389876    5282 generic.go:356] "Generic (PLEG): container finished" podID="393a5585-80b4-4d74-ab5e-51eab898498f" containerID="9be12bf7b96c80d5af59771f6862bf6360914475f9504dea973e8c53b29b51ed" exitCode=0
Nov 03 20:07:56.556793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.389902    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx" event={"ID":"393a5585-80b4-4d74-ab5e-51eab898498f","Type":"ContainerDied","Data":"9be12bf7b96c80d5af59771f6862bf6360914475f9504dea973e8c53b29b51ed"}
Nov 03 20:07:56.557211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.557169    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:07:56.615281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.604243    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:56.736400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.717641    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:56.940285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.933299    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:56.940285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.935304    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:56.944880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.944839    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:56.957095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:56.954549    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:57.016825 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.016722    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:57.017591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.017557    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:57.043219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.027235    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:57.159373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.156724    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:57.313279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.312080    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:57.313279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.290818    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:57.494663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.488956    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:07:57.494663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.489345    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:07:57.650948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.650901    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:07:57.982273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.980551    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:58.114226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:57.988483    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:58.114226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.009295    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:58.114226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.009688    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:58.114226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.009891    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:58.114226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.109579    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:07:58.175412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.175368    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:07:58.182404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.182357    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:58.262631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.262535    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:58.368599 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.368547    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Nov 03 20:07:58.441464 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.438949    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:58.487222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.471491    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"]
Nov 03 20:07:58.487222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.471633    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:07:58.533327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.497443    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz" podUID="5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" containerName="maas-api" containerID="cri-o://832c02ee953995fae6104d8a9cdcea69c279b98c611024f0750f820254c99747" gracePeriod=30
Nov 03 20:07:58.533327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.500364    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:58.533327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.510161    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:58.612520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.612460    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:58.719436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.648319    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:07:58.719436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.715805    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:07:58.739497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.739434    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:07:58.795360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.795273    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"]
Nov 03 20:07:58.825022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.824985    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx" event={"ID":"393a5585-80b4-4d74-ab5e-51eab898498f","Type":"ContainerDied","Data":"906446e11af11793bf7ecf9534e14a5c59c1503684a62db197304888ef9ea2e4"}
Nov 03 20:07:58.825235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.825217    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="906446e11af11793bf7ecf9534e14a5c59c1503684a62db197304888ef9ea2e4"
Nov 03 20:07:58.825556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.825527    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm" podUID="3f3620eb-8848-4b9b-a1bd-085897d9a635" containerName="istio-proxy" containerID="cri-o://149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1" gracePeriod=30
Nov 03 20:07:58.868232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:58.866446    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:07:59.047495 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.047411    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:07:59.072892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.060954    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:07:59.075007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.074961    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:07:59.075595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.075571    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:07:59.169984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.169936    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:59.196223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.173291    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\" (UniqueName: \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\") pod \"393a5585-80b4-4d74-ab5e-51eab898498f\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") "
Nov 03 20:07:59.271221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.270863    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\" (UniqueName: \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\") pod \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") "
Nov 03 20:07:59.271221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.270939    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\") pod \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") "
Nov 03 20:07:59.271221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.270993    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\") pod \"393a5585-80b4-4d74-ab5e-51eab898498f\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") "
Nov 03 20:07:59.339269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.339235    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:07:59.388229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.183269    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:59.525287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.183308    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:07:59.599463 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.515381    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:59.603241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.597967    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln" pod "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" (UID: "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c"). InnerVolumeSpecName "kube-api-access-vczln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:59.636021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.635943    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:07:59.644490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.644450    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:59.659549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.659510    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc" pod "393a5585-80b4-4d74-ab5e-51eab898498f" (UID: "393a5585-80b4-4d74-ab5e-51eab898498f"). InnerVolumeSpecName "kube-api-access-7ktfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:59.702226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.665424    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume" pod "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" (UID: "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:59.702226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.669386    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume" pod "393a5585-80b4-4d74-ab5e-51eab898498f" (UID: "393a5585-80b4-4d74-ab5e-51eab898498f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:59.708219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.704310    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:07:59.904477 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:07:59.894846    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5923c2fe_8bb2_43c4_b1a6_c28f0e9bc970.slice/crio-conmon-832c02ee953995fae6104d8a9cdcea69c279b98c611024f0750f820254c99747.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:07:59.919730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.759173    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:07:59.957757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.942187    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:59.966365    5282 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/393a5585-80b4-4d74-ab5e-51eab898498f/volumes/kubernetes.io~projected/kube-api-access-7ktfc
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.967209    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\" (UniqueName: \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\") pod \"393a5585-80b4-4d74-ab5e-51eab898498f\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") "
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:59.967432    5282 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c/volumes/kubernetes.io~projected/kube-api-access-vczln
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.967468    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln" pod "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" (UID: "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c"). InnerVolumeSpecName "kube-api-access-vczln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.966436    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc" pod "393a5585-80b4-4d74-ab5e-51eab898498f" (UID: "393a5585-80b4-4d74-ab5e-51eab898498f"). InnerVolumeSpecName "kube-api-access-7ktfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.967595    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\" (UniqueName: \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\") pod \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") "
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.967672    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\") pod \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\" (UID: \"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c\") "
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.967759    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\") pod \"393a5585-80b4-4d74-ab5e-51eab898498f\" (UID: \"393a5585-80b4-4d74-ab5e-51eab898498f\") "
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:59.971217    5282 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c/volumes/kubernetes.io~configmap/plugins-config-volume
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.971234    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume" pod "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" (UID: "ead6acea-c6d3-4e38-bedb-dcb9541bcd3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:07:59.971982    5282 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/393a5585-80b4-4d74-ab5e-51eab898498f/volumes/kubernetes.io~configmap/plugins-config-volume
Nov 03 20:07:59.974334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:07:59.971998    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume" pod "393a5585-80b4-4d74-ab5e-51eab898498f" (UID: "393a5585-80b4-4d74-ab5e-51eab898498f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:00.037661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.037622    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktfc\" (UniqueName: \"kubernetes.io/projected/393a5585-80b4-4d74-ab5e-51eab898498f-kube-api-access-7ktfc\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:00.038299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.038266    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vczln\" (UniqueName: \"kubernetes.io/projected/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-kube-api-access-vczln\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:00.038458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.038439    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:00.038593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.038579    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393a5585-80b4-4d74-ab5e-51eab898498f-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:00.081222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.071574    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.097373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.097341    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:08:00.135542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.135511    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:08:00.137083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.118352    5282 generic.go:356] "Generic (PLEG): container finished" podID="5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" containerID="832c02ee953995fae6104d8a9cdcea69c279b98c611024f0750f820254c99747" exitCode=0
Nov 03 20:08:00.137513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.137488    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:08:00.138474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.138452    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz" event={"ID":"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970","Type":"ContainerDied","Data":"832c02ee953995fae6104d8a9cdcea69c279b98c611024f0750f820254c99747"}
Nov 03 20:08:00.264405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.264371    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:08:00.289240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.268304    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:08:00.289569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.289544    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:08:00.363379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.363347    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:08:00.371224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.363631    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:08:00.444222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.371511    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:08:00.541016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.540930    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:08:00.544565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.544537    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:08:00.544760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.249637    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm" podUID="326e6d8b-4ca9-4481-95e1-25aca464e712" containerName="payload-pre-processing" containerID="cri-o://da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735" gracePeriod=25
Nov 03 20:08:00.648490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.646056    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"
Nov 03 20:08:00.648490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.646239    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"
Nov 03 20:08:00.648490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.646320    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2" event={"ID":"ead6acea-c6d3-4e38-bedb-dcb9541bcd3c","Type":"ContainerDied","Data":"5de264d60c7e7dd361c21fef3966767fa17b94e4c41b1c59cc9e6e3e9b12ca9a"}
Nov 03 20:08:00.648490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.646366    5282 scope.go:122] "RemoveContainer" containerID="a1f6f7f8fa3ccfefc169bd5866b87e181e81bc9f007e4978e464bae903a69482"
Nov 03 20:08:00.729087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.729060    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:08:00.834597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.834348    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"]
Nov 03 20:08:00.834597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.834576    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql" podUID="f327a1f3-97da-4c0d-820d-3bd29cee71ca" containerName="maas-api" containerID="cri-o://3d6497c18a03b9c133b881a480397b9cfdd9460b6e80d2eb85e454880e0c00a8" gracePeriod=30
Nov 03 20:08:00.856009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.855755    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-maas-api-tls\") pod \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") "
Nov 03 20:08:00.856009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.855807    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-metrics-certs\") pod \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") "
Nov 03 20:08:00.863384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.856957    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-kube-api-access-bpllj\" (UniqueName: \"kubernetes.io/projected/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-kube-api-access-bpllj\") pod \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") "
Nov 03 20:08:00.863927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.863853    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-openshift-service-ca\") pod \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\" (UID: \"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970\") "
Nov 03 20:08:00.864482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.864430    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-openshift-service-ca" pod "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" (UID: "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:00.866298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.866236    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:00.878072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.877751    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-maas-api-tls" pod "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" (UID: "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:00.878072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.877898    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-kube-api-access-bpllj" pod "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" (UID: "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970"). InnerVolumeSpecName "kube-api-access-bpllj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:00.878072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.877995    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:00.878358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.878143    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-metrics-certs" pod "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" (UID: "5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:00.959629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.957500    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"]
Nov 03 20:08:00.962593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.962516    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c13f7966-5bb7cc5f79-cbvh2"]
Nov 03 20:08:00.974041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.968027    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:00.974041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.968056    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:00.974041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:00.968072    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpllj\" (UniqueName: \"kubernetes.io/projected/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970-kube-api-access-bpllj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:01.020484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.020313    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"]
Nov 03 20:08:01.032752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.032287    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c13f7966-8449f66b8b-j9qlx"]
Nov 03 20:08:01.042114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.040993    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.150378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.149620    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:08:01.172052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.172018    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326e6d8b-4ca9-4481-95e1-25aca464e712-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326e6d8b-4ca9-4481-95e1-25aca464e712-plugins-config-volume\") pod \"326e6d8b-4ca9-4481-95e1-25aca464e712\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") "
Nov 03 20:08:01.172240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.172150    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326e6d8b-4ca9-4481-95e1-25aca464e712-kube-api-access-vdz29\" (UniqueName: \"kubernetes.io/projected/326e6d8b-4ca9-4481-95e1-25aca464e712-kube-api-access-vdz29\") pod \"326e6d8b-4ca9-4481-95e1-25aca464e712\" (UID: \"326e6d8b-4ca9-4481-95e1-25aca464e712\") "
Nov 03 20:08:01.174493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.174434    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326e6d8b-4ca9-4481-95e1-25aca464e712-plugins-config-volume" pod "326e6d8b-4ca9-4481-95e1-25aca464e712" (UID: "326e6d8b-4ca9-4481-95e1-25aca464e712"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:01.178961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.178870    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.284679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.276288    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/326e6d8b-4ca9-4481-95e1-25aca464e712-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:01.284679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.278366    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326e6d8b-4ca9-4481-95e1-25aca464e712-kube-api-access-vdz29" pod "326e6d8b-4ca9-4481-95e1-25aca464e712" (UID: "326e6d8b-4ca9-4481-95e1-25aca464e712"). InnerVolumeSpecName "kube-api-access-vdz29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:01.379402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.379300    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdz29\" (UniqueName: \"kubernetes.io/projected/326e6d8b-4ca9-4481-95e1-25aca464e712-kube-api-access-vdz29\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:01.396577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.390711    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.490788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.490303    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393a5585-80b4-4d74-ab5e-51eab898498f" path="/var/lib/kubelet/pods/393a5585-80b4-4d74-ab5e-51eab898498f/volumes"
Nov 03 20:08:01.492244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.491328    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" path="/var/lib/kubelet/pods/ead6acea-c6d3-4e38-bedb-dcb9541bcd3c/volumes"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.509434    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"]
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510539    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" containerName="payload-pre-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510562    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" containerName="payload-pre-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510589    5282 cpu_manager.go:414] "containerMap: removing container" podUID="326e6d8b-4ca9-4481-95e1-25aca464e712" containerName="payload-pre-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510598    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326e6d8b-4ca9-4481-95e1-25aca464e712" containerName="payload-pre-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510625    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" containerName="maas-api"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510633    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" containerName="maas-api"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510682    5282 cpu_manager.go:414] "containerMap: removing container" podUID="393a5585-80b4-4d74-ab5e-51eab898498f" containerName="payload-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510691    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a5585-80b4-4d74-ab5e-51eab898498f" containerName="payload-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510855    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead6acea-c6d3-4e38-bedb-dcb9541bcd3c" containerName="payload-pre-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510871    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="326e6d8b-4ca9-4481-95e1-25aca464e712" containerName="payload-pre-processing"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510887    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" containerName="maas-api"
Nov 03 20:08:01.515293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.510905    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a5585-80b4-4d74-ab5e-51eab898498f" containerName="payload-processing"
Nov 03 20:08:01.555145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.552185    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.588523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.588413    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"]
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.594656    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.594718    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954ebdee-c21e-404a-a2f3-6b43b0319060-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.594807    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.594990    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6v8s\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-kube-api-access-x6v8s\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.595036    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.595065    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.595117    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.595211    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.596623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.595233    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.641539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.617595    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.670869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.670834    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz" event={"ID":"5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970","Type":"ContainerDied","Data":"c39a522b7760fc5cbfb881a6b7bc75996563f2ec5b8d615ffcecd66a84dd8649"}
Nov 03 20:08:01.671032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.670924    5282 scope.go:122] "RemoveContainer" containerID="832c02ee953995fae6104d8a9cdcea69c279b98c611024f0750f820254c99747"
Nov 03 20:08:01.671271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.671182    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"
Nov 03 20:08:01.676355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.676007    5282 generic.go:356] "Generic (PLEG): container finished" podID="f327a1f3-97da-4c0d-820d-3bd29cee71ca" containerID="3d6497c18a03b9c133b881a480397b9cfdd9460b6e80d2eb85e454880e0c00a8" exitCode=0
Nov 03 20:08:01.676355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.676157    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql" event={"ID":"f327a1f3-97da-4c0d-820d-3bd29cee71ca","Type":"ContainerDied","Data":"3d6497c18a03b9c133b881a480397b9cfdd9460b6e80d2eb85e454880e0c00a8"}
Nov 03 20:08:01.676355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.676213    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql" event={"ID":"f327a1f3-97da-4c0d-820d-3bd29cee71ca","Type":"ContainerDied","Data":"e400b9c2399ce71d11ee930f7d3cce950c1dfbbed7ab3d028eef064c9c1fd6af"}
Nov 03 20:08:01.676355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.676239    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e400b9c2399ce71d11ee930f7d3cce950c1dfbbed7ab3d028eef064c9c1fd6af"
Nov 03 20:08:01.682611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.681539    5282 generic.go:356] "Generic (PLEG): container finished" podID="326e6d8b-4ca9-4481-95e1-25aca464e712" containerID="da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735" exitCode=0
Nov 03 20:08:01.682611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.681658    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm" event={"ID":"326e6d8b-4ca9-4481-95e1-25aca464e712","Type":"ContainerDied","Data":"da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735"}
Nov 03 20:08:01.682611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.681698    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm" event={"ID":"326e6d8b-4ca9-4481-95e1-25aca464e712","Type":"ContainerDied","Data":"55a59bd43e7d2852f7bd2715e900ceca82a81f212e11884c67a3a61b4d88dbea"}
Nov 03 20:08:01.682611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.681767    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"
Nov 03 20:08:01.687359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.687339    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:08:01.690167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.689002    5282 scope.go:122] "RemoveContainer" containerID="da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735"
Nov 03 20:08:01.690721 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:08:01.690698    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:08:01.696276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.696247    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-metrics-certs\") pod \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") "
Nov 03 20:08:01.696380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.696332    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls\") pod \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") "
Nov 03 20:08:01.696438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.696391    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f327a1f3-97da-4c0d-820d-3bd29cee71ca-kube-api-access-562bp\" (UniqueName: \"kubernetes.io/projected/f327a1f3-97da-4c0d-820d-3bd29cee71ca-kube-api-access-562bp\") pod \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") "
Nov 03 20:08:01.696501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.696440    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f327a1f3-97da-4c0d-820d-3bd29cee71ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f327a1f3-97da-4c0d-820d-3bd29cee71ca-openshift-service-ca\") pod \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\" (UID: \"f327a1f3-97da-4c0d-820d-3bd29cee71ca\") "
Nov 03 20:08:01.696905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.696876    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.696993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.696934    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954ebdee-c21e-404a-a2f3-6b43b0319060-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.697262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.697042    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.697339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.697259    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6v8s\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-kube-api-access-x6v8s\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.697339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.697318    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.699312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.697349    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.699400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.699366    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.699472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.699458    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.699535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.699485    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.700268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.700236    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f327a1f3-97da-4c0d-820d-3bd29cee71ca-openshift-service-ca" pod "f327a1f3-97da-4c0d-820d-3bd29cee71ca" (UID: "f327a1f3-97da-4c0d-820d-3bd29cee71ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:01.702751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.701901    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-envoy\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.702751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.702149    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-data\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.702751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.702683    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.703043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.703016    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-credential-socket\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.703169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.703136    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-podinfo\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.703396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.703379    5282 scope.go:122] "RemoveContainer" containerID="da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735"
Nov 03 20:08:01.703458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.703436    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-certs\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.703509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.703476    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"]
Nov 03 20:08:01.703744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.703699    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954ebdee-c21e-404a-a2f3-6b43b0319060-istiod-ca-cert\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.703838 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:08:01.703810    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735\": container with ID starting with da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735 not found: ID does not exist" containerID="da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735"
Nov 03 20:08:01.703890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.703847    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735"} err="failed to get container status \"da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735\": rpc error: code = NotFound desc = could not find container \"da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735\": container with ID starting with da049c89bb8362d5298605c97efeb60b17b022310095bed872969fff6f50b735 not found: ID does not exist"
Nov 03 20:08:01.704069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.704039    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-metrics-certs" pod "f327a1f3-97da-4c0d-820d-3bd29cee71ca" (UID: "f327a1f3-97da-4c0d-820d-3bd29cee71ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:01.704069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.704059    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls" pod "f327a1f3-97da-4c0d-820d-3bd29cee71ca" (UID: "f327a1f3-97da-4c0d-820d-3bd29cee71ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:01.705912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.705862    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f327a1f3-97da-4c0d-820d-3bd29cee71ca-kube-api-access-562bp" pod "f327a1f3-97da-4c0d-820d-3bd29cee71ca" (UID: "f327a1f3-97da-4c0d-820d-3bd29cee71ca"). InnerVolumeSpecName "kube-api-access-562bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:01.706432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.706414    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c13f7966-749f5cc847-7d8sz"]
Nov 03 20:08:01.707896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.707851    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6v8s\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-kube-api-access-x6v8s\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.710069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.710037    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-token\") pod \"e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") " pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:01.715296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.715240    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"]
Nov 03 20:08:01.720374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.720350    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-323cb6ed-659647cd6c-2xlrm"]
Nov 03 20:08:01.760769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.760729    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"]
Nov 03 20:08:01.761545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.761505    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx" podUID="12c2d5d9-4c87-4e8c-8013-8518870b1ed8" containerName="istio-proxy" containerID="cri-o://0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662" gracePeriod=30
Nov 03 20:08:01.808160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.808122    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:01.808386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.808367    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f327a1f3-97da-4c0d-820d-3bd29cee71ca-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:01.812322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.812293    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-562bp\" (UniqueName: \"kubernetes.io/projected/f327a1f3-97da-4c0d-820d-3bd29cee71ca-kube-api-access-562bp\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:01.812440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.812324    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f327a1f3-97da-4c0d-820d-3bd29cee71ca-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:01.822118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.822084    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:01.979282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:01.976528    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:02.184256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.163553    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:08:02.231101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.228803    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.431465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.431179    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.585016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.584885    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:02.586250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.586221    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"]
Nov 03 20:08:02.607572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.607020    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:02.607572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.607119    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:02.607572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.607162    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:02.696109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.696075    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" event={"ID":"954ebdee-c21e-404a-a2f3-6b43b0319060","Type":"ContainerStarted","Data":"121dd77465081628f2ff6deb3f3f0763b9c00265a4b22a2fe3dc67f8fd89a8f9"}
Nov 03 20:08:02.697492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.697468    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"
Nov 03 20:08:02.733917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.733881    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"]
Nov 03 20:08:02.740013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.739975    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-323cb6ed-5f8d4c9695-w7lql"]
Nov 03 20:08:02.933677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.933572    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"]
Nov 03 20:08:02.934758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.934685    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f327a1f3-97da-4c0d-820d-3bd29cee71ca" containerName="maas-api"
Nov 03 20:08:02.934758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.934712    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f327a1f3-97da-4c0d-820d-3bd29cee71ca" containerName="maas-api"
Nov 03 20:08:02.934926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.934894    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f327a1f3-97da-4c0d-820d-3bd29cee71ca" containerName="maas-api"
Nov 03 20:08:02.939231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.939190    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:02.946017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.945949    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ee2323-openshift-default-dockercfg-krfdd\""
Nov 03 20:08:02.972307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:02.971434    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"]
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219274    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6cx\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-kube-api-access-zt6cx\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219327    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219460    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219566    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219605    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219704    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219797    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.219858    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.220004    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.224280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.221545    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322182    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322279    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6cx\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-kube-api-access-zt6cx\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322398    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322555    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322655    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322690    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322765    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322835    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.322893    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.323508    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.337624    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.338050    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.338335    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.338361    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.338662    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.368276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.367390    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:03.376479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.376404    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.385559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.380821    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.444045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.443976    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6cx\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-kube-api-access-zt6cx\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.574396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.565373    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:03.574396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.565709    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326e6d8b-4ca9-4481-95e1-25aca464e712" path="/var/lib/kubelet/pods/326e6d8b-4ca9-4481-95e1-25aca464e712/volumes"
Nov 03 20:08:03.574396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.566520    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970" path="/var/lib/kubelet/pods/5923c2fe-8bb2-43c4-b1a6-c28f0e9bc970/volumes"
Nov 03 20:08:03.574396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.567243    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f327a1f3-97da-4c0d-820d-3bd29cee71ca" path="/var/lib/kubelet/pods/f327a1f3-97da-4c0d-820d-3bd29cee71ca/volumes"
Nov 03 20:08:03.868283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.853116    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" event={"ID":"954ebdee-c21e-404a-a2f3-6b43b0319060","Type":"ContainerStarted","Data":"795e45efd1191c2cc6b1b4746224f3de0141420706c2c2302130a256dd40b941"}
Nov 03 20:08:03.907264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:03.906416    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podStartSLOduration=2.906391222 podStartE2EDuration="2.906391222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:03.887747946 +0000 UTC m=+1676.556949417" watchObservedRunningTime="2026-11-03 20:08:03.906391222 +0000 UTC m=+1676.575592692"
Nov 03 20:08:04.002666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.001361    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:04.003896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.003589    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:08:04.003896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.003641    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 20:08:04.042567 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.042495    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:04.062367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.062322    5282 patch_prober.go:29] interesting pod/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:08:04.062524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.062410    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm" podUID="3f3620eb-8848-4b9b-a1bd-085897d9a635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:08:04.327766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.326889    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:04.327766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.327039    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:04.327766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.327129    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:04.339905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.339852    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"]
Nov 03 20:08:04.973227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.966381    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" event={"ID":"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8","Type":"ContainerStarted","Data":"ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023"}
Nov 03 20:08:04.973227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:04.966465    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" event={"ID":"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8","Type":"ContainerStarted","Data":"7fedd3c84532e6e87fbe96396a10743ea8fcdb753e8e80287b3f6f1b15869e6c"}
Nov 03 20:08:05.009220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:05.001583    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:08:05.009220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:05.001645    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 20:08:05.033269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:05.021297    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podStartSLOduration=3.021274296 podStartE2EDuration="3.021274296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:04.994704088 +0000 UTC m=+1677.663905565" watchObservedRunningTime="2026-11-03 20:08:05.021274296 +0000 UTC m=+1677.690475768"
Nov 03 20:08:05.033807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:05.033657    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:05.569229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:05.568930    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:05.652295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:05.635519    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:05.924811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:05.924652    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:06.009220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:06.005044    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:08:06.009220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:06.005110    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 20:08:06.047237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:06.041865    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:06.073398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:06.067475    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:08:06.684531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:06.683302    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:08:06.684531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:06.683367    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:08:07.004642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:07.004602    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:08:07.004815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:07.004686    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 20:08:07.102117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:07.097169    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:07.519216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:07.519140    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:07.662074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:07.661698    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:08.044757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.001692    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:08:08.044757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.001799    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 20:08:08.371338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.340765    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:08.819246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.819144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:08.826370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.826323    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"]
Nov 03 20:08:08.851733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.851397    5282 patch_prober.go:29] interesting pod/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:08:08.851733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.851453    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx" podUID="12c2d5d9-4c87-4e8c-8013-8518870b1ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:08:08.913907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.911399    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"]
Nov 03 20:08:08.913907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.911447    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"]
Nov 03 20:08:08.913907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.912794    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:08.919730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.919700    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-ee2323\""
Nov 03 20:08:08.932945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.932906    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"]
Nov 03 20:08:08.933069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.933044    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:08.939494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.939465    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-ee2323-dockercfg-lvtqq\""
Nov 03 20:08:09.003425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:08.963030    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.015443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.004594    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:08:09.015443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.004653    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 20:08:09.082282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.058720    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw2rc\" (UniqueName: \"kubernetes.io/projected/b7156169-be29-4e7b-b323-6f18e4e09ba9-kube-api-access-lw2rc\") pod \"payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:09.082282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.058788    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:09.082282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.058917    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5xz\" (UniqueName: \"kubernetes.io/projected/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-kube-api-access-fv5xz\") pod \"payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:09.082282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.058977    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7156169-be29-4e7b-b323-6f18e4e09ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.165443    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw2rc\" (UniqueName: \"kubernetes.io/projected/b7156169-be29-4e7b-b323-6f18e4e09ba9-kube-api-access-lw2rc\") pod \"payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.165526    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.165716    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5xz\" (UniqueName: \"kubernetes.io/projected/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-kube-api-access-fv5xz\") pod \"payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.165795    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7156169-be29-4e7b-b323-6f18e4e09ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.166808    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7156169-be29-4e7b-b323-6f18e4e09ba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.231004    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.420949    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw2rc\" (UniqueName: \"kubernetes.io/projected/b7156169-be29-4e7b-b323-6f18e4e09ba9-kube-api-access-lw2rc\") pod \"payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:09.427324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.421704    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5xz\" (UniqueName: \"kubernetes.io/projected/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-kube-api-access-fv5xz\") pod \"payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:09.440233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.439659    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:09.441287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.441144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.608083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.608044    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:09.663371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.663337    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:09.971831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:09.971795    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"]
Nov 03 20:08:10.034932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:10.034292    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:08:10.034932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:10.034356    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 20:08:10.129545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:10.061572    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:10.215979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:10.214829    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"]
Nov 03 20:08:10.266851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:10.266782    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2" event={"ID":"b7156169-be29-4e7b-b323-6f18e4e09ba9","Type":"ContainerStarted","Data":"762b28008966e4072be9f568a80b6a7facbb326e25868a36293d060f837b3d13"}
Nov 03 20:08:10.725443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:10.725349    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:11.006746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.006708    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:11.007648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.007508    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:11.008343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.008324    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:08:11.275732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.275635    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2" event={"ID":"b7156169-be29-4e7b-b323-6f18e4e09ba9","Type":"ContainerStarted","Data":"76c169b9736b517d19e7cfc9965a3ce8d2d30f8b7884c86e423c53ba7b6d1481"}
Nov 03 20:08:11.275732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.275702    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:11.277605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.277577    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt" event={"ID":"321af73b-d2a5-4e37-9ee6-a6f75d0b263c","Type":"ContainerStarted","Data":"cbc9b75df839dee6b5cbccb8bccaec06142139895082b651e9b6449cabeab38f"}
Nov 03 20:08:11.277746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.277623    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt" event={"ID":"321af73b-d2a5-4e37-9ee6-a6f75d0b263c","Type":"ContainerStarted","Data":"d3c4f14b2a7c2755449cfb170ee08686144ee064917290b62c64f1e9dd40e742"}
Nov 03 20:08:11.294450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.294393    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2" podStartSLOduration=3.294376662 podStartE2EDuration="3.294376662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:11.290621705 +0000 UTC m=+1683.959823176" watchObservedRunningTime="2026-11-03 20:08:11.294376662 +0000 UTC m=+1683.963578131"
Nov 03 20:08:11.317770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.307542    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt" podStartSLOduration=3.307528147 podStartE2EDuration="3.307528147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:11.305806981 +0000 UTC m=+1683.975008451" watchObservedRunningTime="2026-11-03 20:08:11.307528147 +0000 UTC m=+1683.976729647"
Nov 03 20:08:12.591187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.481428    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:12.591187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:11.866659    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:12.591187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.247073    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:12.591187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.335044    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:12.804642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.802863    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:08:12.876524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876490    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-envoy\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.876667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876551    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-certs\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.876737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876662    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-credential-socket\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.876806 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876789    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-podinfo\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.876868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876803    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-certs" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:12.876868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876842    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-kube-api-access-f7lct\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-kube-api-access-f7lct\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.876974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876897    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-socket\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.876974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.876930    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-token\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-token\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.877276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.877246    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-data\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.877435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.877363    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istiod-ca-cert\") pod \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\" (UID: \"12c2d5d9-4c87-4e8c-8013-8518870b1ed8\") "
Nov 03 20:08:12.877509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.877162    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-socket" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:12.877575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.877524    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-data" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:12.877575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.877547    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-credential-socket" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:12.878868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.878623    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.878868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.878652    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.878868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.878667    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.878868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.878681    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.878868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.878840    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istiod-ca-cert" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:12.879425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.879401    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-token" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:12.885766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.885740    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-kube-api-access-f7lct" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "kube-api-access-f7lct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:12.885886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.885864    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-envoy" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:12.887767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.887734    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-podinfo" pod "12c2d5d9-4c87-4e8c-8013-8518870b1ed8" (UID: "12c2d5d9-4c87-4e8c-8013-8518870b1ed8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:12.980580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.980540    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.980580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.980582    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.980804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.980623    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7lct\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-kube-api-access-f7lct\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.980804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.980638    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:12.980804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:12.980652    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12c2d5d9-4c87-4e8c-8013-8518870b1ed8-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.010036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.010004    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:13.093407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.091991    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.184677    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-certs\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.184766    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-envoy\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.184796    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3620eb-8848-4b9b-a1bd-085897d9a635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3620eb-8848-4b9b-a1bd-085897d9a635-istiod-ca-cert\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.184820    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-token\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-token\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.184845    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-socket\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.184875    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-kube-api-access-w4xlp\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-kube-api-access-w4xlp\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.184979    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-podinfo\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.185001    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-credential-socket\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.185273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.185064    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-data\") pod \"3f3620eb-8848-4b9b-a1bd-085897d9a635\" (UID: \"3f3620eb-8848-4b9b-a1bd-085897d9a635\") "
Nov 03 20:08:13.189273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.185801    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-data" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:13.189273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.185967    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-certs" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:13.189273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.188039    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-envoy" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:13.189273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.188363    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3620eb-8848-4b9b-a1bd-085897d9a635-istiod-ca-cert" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:13.189273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.188646    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-credential-socket" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:13.189645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.189424    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-socket" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:13.192291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.191326    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-kube-api-access-w4xlp" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "kube-api-access-w4xlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:13.192719 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.192692    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-token" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:13.193068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.193051    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-podinfo" pod "3f3620eb-8848-4b9b-a1bd-085897d9a635" (UID: "3f3620eb-8848-4b9b-a1bd-085897d9a635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:13.287398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287361    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3620eb-8848-4b9b-a1bd-085897d9a635-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287392    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287403    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287412    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4xlp\" (UniqueName: \"kubernetes.io/projected/3f3620eb-8848-4b9b-a1bd-085897d9a635-kube-api-access-w4xlp\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287423    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287431    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287439    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287449    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.287630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.287457    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3620eb-8848-4b9b-a1bd-085897d9a635-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:13.329973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.329934    5282 generic.go:356] "Generic (PLEG): container finished" podID="12c2d5d9-4c87-4e8c-8013-8518870b1ed8" containerID="0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662" exitCode=0
Nov 03 20:08:13.330143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.329990    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"
Nov 03 20:08:13.330143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.330015    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx" event={"ID":"12c2d5d9-4c87-4e8c-8013-8518870b1ed8","Type":"ContainerDied","Data":"0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662"}
Nov 03 20:08:13.330143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.330072    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx" event={"ID":"12c2d5d9-4c87-4e8c-8013-8518870b1ed8","Type":"ContainerDied","Data":"d96cf38c59315067309aea49d6594e0b2c0649bbd4f41f46519b91663737a0b5"}
Nov 03 20:08:13.330143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.330103    5282 scope.go:122] "RemoveContainer" containerID="0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662"
Nov 03 20:08:13.331480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.331457    5282 generic.go:356] "Generic (PLEG): container finished" podID="3f3620eb-8848-4b9b-a1bd-085897d9a635" containerID="149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1" exitCode=0
Nov 03 20:08:13.331598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.331486    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm" event={"ID":"3f3620eb-8848-4b9b-a1bd-085897d9a635","Type":"ContainerDied","Data":"149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1"}
Nov 03 20:08:13.331598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.331505    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"
Nov 03 20:08:13.331598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.331546    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm" event={"ID":"3f3620eb-8848-4b9b-a1bd-085897d9a635","Type":"ContainerDied","Data":"7ae340e656a6162e5721f1e18d4e4f2a98ca01b8e223b059070a329b7cb62227"}
Nov 03 20:08:13.333518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.333490    5282 generic.go:356] "Generic (PLEG): container finished" podID="dfbfc761-e296-4114-b4e3-da2df1301934" containerID="1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5" exitCode=0
Nov 03 20:08:13.333651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.333560    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" event={"ID":"dfbfc761-e296-4114-b4e3-da2df1301934","Type":"ContainerDied","Data":"1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5"}
Nov 03 20:08:13.334077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.334050    5282 scope.go:122] "RemoveContainer" containerID="1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5"
Nov 03 20:08:13.335566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.335537    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:13.335676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.335603    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:13.335676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.335635    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:13.346593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.346575    5282 scope.go:122] "RemoveContainer" containerID="0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662"
Nov 03 20:08:13.346879 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:08:13.346854    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662\": container with ID starting with 0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662 not found: ID does not exist" containerID="0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662"
Nov 03 20:08:13.346966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.346889    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662"} err="failed to get container status \"0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662\": rpc error: code = NotFound desc = could not find container \"0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662\": container with ID starting with 0f5dd6645aaed58abdc18b0fd8ae51eddaf9cc71ea4ab1e33c3450d92973a662 not found: ID does not exist"
Nov 03 20:08:13.346966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.346912    5282 scope.go:122] "RemoveContainer" containerID="149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1"
Nov 03 20:08:13.357338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.357315    5282 scope.go:122] "RemoveContainer" containerID="149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1"
Nov 03 20:08:13.357626 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:08:13.357605    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1\": container with ID starting with 149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1 not found: ID does not exist" containerID="149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1"
Nov 03 20:08:13.357707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.357637    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1"} err="failed to get container status \"149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1\": rpc error: code = NotFound desc = could not find container \"149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1\": container with ID starting with 149189eb2dab3e9de9b8de7f8c31e4d0ff72d9c16658b1774b3bfc935de2f6b1 not found: ID does not exist"
Nov 03 20:08:13.376709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.376684    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"]
Nov 03 20:08:13.386573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.386548    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c13f7966-gw-openshift-default-7b8777989c-ndwkm"]
Nov 03 20:08:13.401526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.399148    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"]
Nov 03 20:08:13.404604 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.403901    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-323cb6ed-openshift-default-7959d7f96f-rk7lx"]
Nov 03 20:08:13.457189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.457122    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c2d5d9-4c87-4e8c-8013-8518870b1ed8" path="/var/lib/kubelet/pods/12c2d5d9-4c87-4e8c-8013-8518870b1ed8/volumes"
Nov 03 20:08:13.457755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.457739    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3620eb-8848-4b9b-a1bd-085897d9a635" path="/var/lib/kubelet/pods/3f3620eb-8848-4b9b-a1bd-085897d9a635/volumes"
Nov 03 20:08:13.709022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.708944    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:13.893279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:13.893240    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:14.253677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.253638    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"]
Nov 03 20:08:14.254839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.254810    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3f3620eb-8848-4b9b-a1bd-085897d9a635" containerName="istio-proxy"
Nov 03 20:08:14.254839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.254837    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3620eb-8848-4b9b-a1bd-085897d9a635" containerName="istio-proxy"
Nov 03 20:08:14.255030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.254863    5282 cpu_manager.go:414] "containerMap: removing container" podUID="12c2d5d9-4c87-4e8c-8013-8518870b1ed8" containerName="istio-proxy"
Nov 03 20:08:14.255030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.254871    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c2d5d9-4c87-4e8c-8013-8518870b1ed8" containerName="istio-proxy"
Nov 03 20:08:14.255135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.255039    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3620eb-8848-4b9b-a1bd-085897d9a635" containerName="istio-proxy"
Nov 03 20:08:14.255135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.255053    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c2d5d9-4c87-4e8c-8013-8518870b1ed8" containerName="istio-proxy"
Nov 03 20:08:14.260882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.260855    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.267834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.267802    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-acd98d21-openshift-default-dockercfg-b8v8z\""
Nov 03 20:08:14.273674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.273649    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"]
Nov 03 20:08:14.307444 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307403    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.307635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307478    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.307635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307501    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.307752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307676    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.307752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307727    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkvh\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-kube-api-access-8qkvh\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.307852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307800    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.307907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307873    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.307998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.307971    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.308100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.308083    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.316040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.316010    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:14.347692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.347663    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" event={"ID":"dfbfc761-e296-4114-b4e3-da2df1301934","Type":"ContainerStarted","Data":"e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d"}
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438288    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438363    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438396    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438517    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438577    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkvh\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-kube-api-access-8qkvh\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438616    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438727    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438801    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.444253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.438880    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.457229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.454042    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.457229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.454423    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-data\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.457229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.454680    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.459190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.457893    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-credential-socket\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.459190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.458161    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-certs\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.484319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.484284    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:14.512283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.502138    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-podinfo\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.512793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.512764    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-envoy\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.513569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.513545    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-token\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.514379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.514317    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkvh\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-kube-api-access-8qkvh\") pod \"e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") " pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.580566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.580524    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:14.685221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:14.671820    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.080981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.080946    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:08:15.081655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.081629    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:08:15.081858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.081831    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 20:08:15.303993 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:08:15.303960    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28e0e71_7687_4f07_a1e0_4d6382ac4d74.slice/crio-f07e1bc4a352e64fd7faf2124c6096b59f4273887f5591b765b583269e09ea22 WatchSource:0}: Error finding container f07e1bc4a352e64fd7faf2124c6096b59f4273887f5591b765b583269e09ea22: Status 404 returned error can't find the container with id f07e1bc4a352e64fd7faf2124c6096b59f4273887f5591b765b583269e09ea22
Nov 03 20:08:15.305167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.305141    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"]
Nov 03 20:08:15.306812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.306775    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:15.306921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.306874    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:15.306980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.306920    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:08:15.335459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.335431    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"]
Nov 03 20:08:15.341003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.340981    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.347035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.347008    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-ee2323-serving-cert\""
Nov 03 20:08:15.352928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.352900    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"]
Nov 03 20:08:15.367845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.367814    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" event={"ID":"f28e0e71-7687-4f07-a1e0-4d6382ac4d74","Type":"ContainerStarted","Data":"f07e1bc4a352e64fd7faf2124c6096b59f4273887f5591b765b583269e09ea22"}
Nov 03 20:08:15.409565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.409537    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.424473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.424446    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.424620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.424596    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rlg\" (UniqueName: \"kubernetes.io/projected/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-kube-api-access-h5rlg\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.424693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.424668    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.424852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.424829    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.527223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.527054    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.527437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.527319    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.527528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.527498    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rlg\" (UniqueName: \"kubernetes.io/projected/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-kube-api-access-h5rlg\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.560946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.527780    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.560946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.531561    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.560946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.557222    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.564504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.564430    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.566764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.566674    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rlg\" (UniqueName: \"kubernetes.io/projected/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-kube-api-access-h5rlg\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.589305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.583813    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:15.717617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.715868    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:15.717617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:15.716175    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:16.047319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.047278    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:08:16.047577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.047553    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 20:08:16.211658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.209228    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:08:16.213756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.213725    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"]
Nov 03 20:08:16.295343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.293880    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:16.436642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.436540    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7" event={"ID":"20a4a27d-6002-43e8-bfa5-af8a681cbf5d","Type":"ContainerStarted","Data":"22a0f52e76dd4031cbddacee98bcc568a9844830fb4ea697e73cedcd9bca4581"}
Nov 03 20:08:16.439823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.439786    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" event={"ID":"f28e0e71-7687-4f07-a1e0-4d6382ac4d74","Type":"ContainerStarted","Data":"0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767"}
Nov 03 20:08:16.630249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.620967    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:16.663812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.663767    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:16.701888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:16.701480    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podStartSLOduration=2.701456236 podStartE2EDuration="2.701456236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:16.493898315 +0000 UTC m=+1689.163099785" watchObservedRunningTime="2026-11-03 20:08:16.701456236 +0000 UTC m=+1689.370657706"
Nov 03 20:08:17.023411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.023345    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:17.028265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.028238    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"]
Nov 03 20:08:17.031790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.031770    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c823108-g5hh6"]
Nov 03 20:08:17.046955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.046918    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:08:17.047213 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.047168    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 20:08:17.464107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.464075    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8" path="/var/lib/kubelet/pods/6c2e2a4d-88c9-4af1-9bb1-3b3bbd3918b8/volumes"
Nov 03 20:08:17.464530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.464413    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:17.464530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.464466    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:08:17.464530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.464482    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7" event={"ID":"20a4a27d-6002-43e8-bfa5-af8a681cbf5d","Type":"ContainerStarted","Data":"8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577"}
Nov 03 20:08:17.499283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.499216    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7" podStartSLOduration=1.8981835299999998 podStartE2EDuration="2.499180978s" totalImagesPullingTime="600.997448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:17.498560951 +0000 UTC m=+1690.167762439" watchObservedRunningTime="2026-11-03 20:08:17.499180978 +0000 UTC m=+1690.168382448"
Nov 03 20:08:17.755108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:17.755060    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:18.044744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:18.044647    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:08:18.044744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:18.044701    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 20:08:18.464158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:18.464075    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:18.511953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:18.511924    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:18.745938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:18.745908    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:18.888941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:18.888900    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:19.045231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.045104    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:08:19.045231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.045164    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 20:08:19.299750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.299664    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:19.937562 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.937531    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:19.958132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.958097    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"]
Nov 03 20:08:19.962597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.962577    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:19.990229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.986155    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-acd98d21\""
Nov 03 20:08:20.004219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:19.994818    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"]
Nov 03 20:08:20.017997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.017963    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"]
Nov 03 20:08:20.026310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.026280    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.035887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.035864    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-acd98d21-dockercfg-z9pnt\""
Nov 03 20:08:20.045080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.045058    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:08:20.045209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.045097    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 20:08:20.045560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.045535    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"]
Nov 03 20:08:20.130218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.129896    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hrb\" (UniqueName: \"kubernetes.io/projected/c8bddb75-dbbc-4907-8c40-0ca54c150a92-kube-api-access-x2hrb\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:20.130218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.130143    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsh8\" (UniqueName: \"kubernetes.io/projected/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-kube-api-access-xxsh8\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.130447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.130284    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.130447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.130328    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bddb75-dbbc-4907-8c40-0ca54c150a92-plugins-config-volume\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:20.144215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.143485    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:20.254221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.250613    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.254221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.252847    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.254221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.252922    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bddb75-dbbc-4907-8c40-0ca54c150a92-plugins-config-volume\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:20.254221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.253099    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hrb\" (UniqueName: \"kubernetes.io/projected/c8bddb75-dbbc-4907-8c40-0ca54c150a92-kube-api-access-x2hrb\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:20.254221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.253371    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsh8\" (UniqueName: \"kubernetes.io/projected/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-kube-api-access-xxsh8\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.254644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.254590    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bddb75-dbbc-4907-8c40-0ca54c150a92-plugins-config-volume\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:20.335234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.334911    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hrb\" (UniqueName: \"kubernetes.io/projected/c8bddb75-dbbc-4907-8c40-0ca54c150a92-kube-api-access-x2hrb\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:20.355220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.342923    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsh8\" (UniqueName: \"kubernetes.io/projected/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-kube-api-access-xxsh8\") pod \"payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.362220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.356512    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:20.613274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.612376    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:20.629860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.629335    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:08:20.655641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.655606    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:20.812266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.810689    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:20.901285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.901255    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"]
Nov 03 20:08:20.999217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:20.998832    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"]
Nov 03 20:08:21.044828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.044797    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 03 20:08:21.044913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.044851    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 03 20:08:21.549021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.548986    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk" event={"ID":"c8bddb75-dbbc-4907-8c40-0ca54c150a92","Type":"ContainerStarted","Data":"24d24fca98d790832eea4f7c978ed10291b9e91fba5380f364596c2f56d4c9c6"}
Nov 03 20:08:21.549186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.549045    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk" event={"ID":"c8bddb75-dbbc-4907-8c40-0ca54c150a92","Type":"ContainerStarted","Data":"977a2ba78ba9e6843cbc23563572f5a5c49ecd127c466d5a38f24b98c3521d27"}
Nov 03 20:08:21.549266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.549228    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:21.550669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.550642    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx" event={"ID":"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338","Type":"ContainerStarted","Data":"7a571410d3c100d550b049d1d7ba829233f1f98060bc8cc447bae4bc1a485aa9"}
Nov 03 20:08:21.550776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.550690    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx" event={"ID":"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338","Type":"ContainerStarted","Data":"fb287f3e22e0185c6b59a52ae284fd0239c17c963c327958361c2d5bccdbdb8c"}
Nov 03 20:08:21.550776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.550732    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:21.571005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.570962    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk" podStartSLOduration=2.570950424 podStartE2EDuration="2.570950424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:21.566816322 +0000 UTC m=+1694.236017792" watchObservedRunningTime="2026-11-03 20:08:21.570950424 +0000 UTC m=+1694.240151903"
Nov 03 20:08:21.589961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.589922    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx" podStartSLOduration=2.589908973 podStartE2EDuration="2.589908973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:21.588283369 +0000 UTC m=+1694.257484837" watchObservedRunningTime="2026-11-03 20:08:21.589908973 +0000 UTC m=+1694.259110447"
Nov 03 20:08:21.753251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.743943    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:21.912836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:21.912749    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:08:22.175303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:22.168569    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:08:22.236392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:22.227216    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:08:22.372744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:22.367777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:08:22.780246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:22.780186    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:23.361223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:23.360458    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:08:23.839234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:23.839181    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:24.499392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:24.499359    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:08:24.929773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:24.929689    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:26.398334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:26.398293    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:08:26.398990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:26.398958    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:08:26.807132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:26.806181    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:27.487412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:27.487372    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:27.924101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:27.924008    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.351981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:28.351949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.700629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:28.700538    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:28.889821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:28.889789    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:30.042048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:30.042018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:30.160017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:30.159978    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:30.314098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:30.314017    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:31.351183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:31.351049    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:31.630340 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:08:31.630243    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:08:32.268116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:32.268083    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:32.564568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:32.564478    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:08:32.565006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:32.564811    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:08:33.253504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:33.253470    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:33.911721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:33.911686    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:34.043995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:34.043965    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:34.402888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:34.402852    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:35.133173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.133074    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:35.168161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.168127    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"]
Nov 03 20:08:35.173002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.172974    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.181597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.181561    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-0ffd89\""
Nov 03 20:08:35.181751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.181718    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-0ffd89-dockercfg-hrw2t\""
Nov 03 20:08:35.188087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.188059    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"]
Nov 03 20:08:35.204823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.204784    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9g5j\" (UniqueName: \"kubernetes.io/projected/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-kube-api-access-s9g5j\") pod \"payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.205045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.205025    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.313820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.313748    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:35.316113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.316068    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9g5j\" (UniqueName: \"kubernetes.io/projected/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-kube-api-access-s9g5j\") pod \"payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.340480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.340449    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.343451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.343051    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.371093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.371060    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9g5j\" (UniqueName: \"kubernetes.io/projected/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-kube-api-access-s9g5j\") pod \"payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.500618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.500579    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:35.964445 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:08:35.964414    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2ceb5a_f3de_40c5_91ba_eed7d0c374ac.slice/crio-d51c833745269aa1686221eb672db5c5cf7f48e79d1147694834f8f8e679b86b WatchSource:0}: Error finding container d51c833745269aa1686221eb672db5c5cf7f48e79d1147694834f8f8e679b86b: Status 404 returned error can't find the container with id d51c833745269aa1686221eb672db5c5cf7f48e79d1147694834f8f8e679b86b
Nov 03 20:08:35.964679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:35.964655    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"]
Nov 03 20:08:36.006031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:36.005999    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt" event={"ID":"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac","Type":"ContainerStarted","Data":"d51c833745269aa1686221eb672db5c5cf7f48e79d1147694834f8f8e679b86b"}
Nov 03 20:08:36.025476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:36.025442    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:36.724652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:36.724618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:37.044257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:37.041813    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt" event={"ID":"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac","Type":"ContainerStarted","Data":"36aad6d9f4b1c19df0832a6cd7bc8539037852e7b511fc88de0052932ffa01e9"}
Nov 03 20:08:37.044257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:37.043927    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:37.074413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:37.067860    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt" podStartSLOduration=2.06784523 podStartE2EDuration="2.06784523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:37.058565016 +0000 UTC m=+1709.727766488" watchObservedRunningTime="2026-11-03 20:08:37.06784523 +0000 UTC m=+1709.737046737"
Nov 03 20:08:37.529641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:37.529610    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:38.279855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:38.279825    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:38.994136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:38.994099    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:39.827360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:39.827324    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:40.668377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:40.668333    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:41.291973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:41.291927    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:41.481633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:41.481602    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:42.030501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:42.030467    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:42.934519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:42.934483    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:43.703433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.703395    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:43.712356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.712314    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"]
Nov 03 20:08:43.718767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.718741    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:43.730008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.729566    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-0ffd89-serving-cert\""
Nov 03 20:08:43.740835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.740803    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"]
Nov 03 20:08:43.829871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.829833    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:43.899222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.897493    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbd6m\" (UniqueName: \"kubernetes.io/projected/0972f4db-9691-4eb9-94a2-5dd461e656dd-kube-api-access-wbd6m\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:43.899222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.897570    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:43.899222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.897736    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:43.899222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.898026    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0972f4db-9691-4eb9-94a2-5dd461e656dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:43.989677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:43.989643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:44.001320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.001286    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0972f4db-9691-4eb9-94a2-5dd461e656dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.002462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.002422    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbd6m\" (UniqueName: \"kubernetes.io/projected/0972f4db-9691-4eb9-94a2-5dd461e656dd-kube-api-access-wbd6m\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.002670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.002650    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.002933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.002919    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.003448 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:08:44.003425    5282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-serving-cert: secret "maas-api-e2e-shared-b-w6-0ffd89-serving-cert" not found
Nov 03 20:08:44.003619 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:08:44.003609    5282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls podName:0972f4db-9691-4eb9-94a2-5dd461e656dd nodeName:}" failed. No retries permitted until 2026-11-03 20:08:44.503587377 +0000 UTC m=+1717.172788824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls") pod "maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c" (UID: "0972f4db-9691-4eb9-94a2-5dd461e656dd") : secret "maas-api-e2e-shared-b-w6-0ffd89-serving-cert" not found
Nov 03 20:08:44.008626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.002298    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0972f4db-9691-4eb9-94a2-5dd461e656dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.017924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.017893    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.027656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.025608    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbd6m\" (UniqueName: \"kubernetes.io/projected/0972f4db-9691-4eb9-94a2-5dd461e656dd-kube-api-access-wbd6m\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.468127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.468043    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:44.526398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.526353    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.530902 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.530843    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.649223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.644274    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:44.915684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.915649    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:44.966693 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:08:44.966662    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0972f4db_9691_4eb9_94a2_5dd461e656dd.slice/crio-2ea4a99c8bbbc308ffcdfb3113ab3627eec6adfd17662b9dd9a9a2e5181bf4f0 WatchSource:0}: Error finding container 2ea4a99c8bbbc308ffcdfb3113ab3627eec6adfd17662b9dd9a9a2e5181bf4f0: Status 404 returned error can't find the container with id 2ea4a99c8bbbc308ffcdfb3113ab3627eec6adfd17662b9dd9a9a2e5181bf4f0
Nov 03 20:08:44.966807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:44.966719    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"]
Nov 03 20:08:45.220221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:45.218053    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c" event={"ID":"0972f4db-9691-4eb9-94a2-5dd461e656dd","Type":"ContainerStarted","Data":"2ea4a99c8bbbc308ffcdfb3113ab3627eec6adfd17662b9dd9a9a2e5181bf4f0"}
Nov 03 20:08:45.732473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:45.732392    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:45.974466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:45.974416    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:46.296757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:46.296714    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c" event={"ID":"0972f4db-9691-4eb9-94a2-5dd461e656dd","Type":"ContainerStarted","Data":"0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f"}
Nov 03 20:08:46.298813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:46.298788    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:46.355490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:46.355426    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c" podStartSLOduration=2.8423647880000003 podStartE2EDuration="3.355406727s" totalImagesPullingTime="513.041939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:46.354481263 +0000 UTC m=+1719.023682733" watchObservedRunningTime="2026-11-03 20:08:46.355406727 +0000 UTC m=+1719.024608200"
Nov 03 20:08:46.719340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:46.719232    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:47.409432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:47.409392    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:48.320559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:48.320521    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:48.998877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:48.998846    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:49.089748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:49.089681    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:08:49.754356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:49.754312    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:50.359491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:50.359440    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:50.500779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:50.500741    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.193227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:51.191552    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.712016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:51.711985    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:51.823768 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:51.823735    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:52.111038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:52.111005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:52.889253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:52.887157    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:53.356577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:53.356540    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:08:53.722459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:53.722366    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:54.516074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:54.516036    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:55.422681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:55.422645    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:56.205318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:56.205281    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:56.947895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:56.943769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:57.599294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:57.599243    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:57.734413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:57.734353    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:58.502336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:58.502304    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:59.162809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:59.162777    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:59.287009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:59.285860    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:08:59.902479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:08:59.902448    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:00.631467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:00.631437    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:01.325159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:01.325121    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:01.618417 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:09:01.618337    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:09:01.664975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:01.664938    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:09:01.944965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:01.944858    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.331881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.331844    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"]
Nov 03 20:09:02.337899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.337873    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.349717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.349690    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d441dd-openshift-default-dockercfg-x62gq\""
Nov 03 20:09:02.350192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.350159    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"]
Nov 03 20:09:02.373114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373078    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-data\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373142    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-token\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373310    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-socket\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373456    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-podinfo\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373512    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42k2\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-kube-api-access-l42k2\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373616    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-envoy\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373734    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-credential-socket\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373785    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-certs\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.373857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.373843    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istiod-ca-cert\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.378607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.378572    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.535279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.498988    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-data\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.569389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.569356    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-token\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.569718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.569694    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-socket\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.569943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.569927    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-podinfo\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.570280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.526260    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-data\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.570280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.570077    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42k2\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-kube-api-access-l42k2\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.585290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.570549    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-envoy\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.585290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.570737    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-credential-socket\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.585290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.570818    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-certs\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.585290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.570912    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istiod-ca-cert\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.585290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.571785    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istiod-ca-cert\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.585290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.572053    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-socket\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.587052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.587026    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-credential-socket\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.587557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.587538    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-certs\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.609648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.609487    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.627774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.624805    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-podinfo\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.627774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.625665    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-token\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.652243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.648124    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-envoy\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.655705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.653754    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42k2\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-kube-api-access-l42k2\") pod \"e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") " pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.671518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.671253    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:02.763241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.761769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:02.891914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:02.883747    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:03.222810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:03.222768    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:03.222932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:03.222879    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:03.222932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:03.222922    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:03.225300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:03.225274    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"]
Nov 03 20:09:03.421268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:03.418810    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:03.770853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:03.770808    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" event={"ID":"2e13f182-06e1-4122-922b-ebc4ebeca7ce","Type":"ContainerStarted","Data":"773e386b3e898f7d23d056b40466205291ad5913f6ade00184935326b1639941"}
Nov 03 20:09:03.771484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:03.771457    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" event={"ID":"2e13f182-06e1-4122-922b-ebc4ebeca7ce","Type":"ContainerStarted","Data":"16719a2696d87382ef04375e76681e593674e6706481cc257f5112e313432025"}
Nov 03 20:09:04.725226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:04.724506    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:04.748159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:04.748116    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:04.839226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:04.833730    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:04.852373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:04.848043    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podStartSLOduration=2.848021727 podStartE2EDuration="2.848021727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:03.83487523 +0000 UTC m=+1736.504076699" watchObservedRunningTime="2026-11-03 20:09:04.848021727 +0000 UTC m=+1737.517223197"
Nov 03 20:09:04.864225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:04.856470    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:05.249311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.246813    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:05.398225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.398015    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:05.662873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.662772    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:05.695219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.694758    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:09:05.706410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.706356    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:05.731231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.726875    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:09:05.744228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.743917    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:09:05.775231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.768301    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:05.961279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:05.960326    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:06.004065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:06.004002    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:06.024380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:06.024336    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:06.135162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:06.130323    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:06.135162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:06.130714    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:09:06.264229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:06.257509    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:06.600152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:06.600052    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:06.738093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:06.738056    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:07.555689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:07.555661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:07.883305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:07.883270    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:08.019348 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:08.019312    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:08.827136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:08.827100    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:09.833684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:09.833647    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:10.595011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:10.594972    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:11.304012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:11.303974    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:12.004865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:12.004834    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:12.793725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:12.793682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:13.491455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:13.491421    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:14.190060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:14.190014    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:14.837962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:14.837937    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:15.030236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:15.030142    5282 generic.go:356] "Generic (PLEG): container finished" podID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerID="0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767" exitCode=0
Nov 03 20:09:15.030236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:15.030189    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" event={"ID":"f28e0e71-7687-4f07-a1e0-4d6382ac4d74","Type":"ContainerDied","Data":"0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767"}
Nov 03 20:09:15.030704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:15.030683    5282 scope.go:122] "RemoveContainer" containerID="0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767"
Nov 03 20:09:15.032504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:15.032473    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:15.032578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:15.032552    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:15.032619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:15.032589    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:15.507385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:15.507349    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:16.039692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:16.039651    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" event={"ID":"f28e0e71-7687-4f07-a1e0-4d6382ac4d74","Type":"ContainerStarted","Data":"49eec8224f9ca146062d29d3fb56def448f393edf0346f9a33a1b82d8c49d6a1"}
Nov 03 20:09:16.236364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:16.236321    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:16.598223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:16.597276    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:09:16.599912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:16.599887    5282 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:09:16.600056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:16.599948    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 20:09:17.037777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.037744    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"]
Nov 03 20:09:17.043063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.043039    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.052531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.052423    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d441dd-serving-cert\""
Nov 03 20:09:17.063948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.063922    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"]
Nov 03 20:09:17.117953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.117912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:17.218053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.218005    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-metrics-certs\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.218413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.218384    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f211138-01d6-408c-a6c8-fe61cb097aae-openshift-service-ca\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.225570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.225502    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-maas-api-tls\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.230211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.226059    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbqdz\" (UniqueName: \"kubernetes.io/projected/5f211138-01d6-408c-a6c8-fe61cb097aae-kube-api-access-qbqdz\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.239221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.231889    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:17.345273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.343827    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-maas-api-tls\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.345273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.344726    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbqdz\" (UniqueName: \"kubernetes.io/projected/5f211138-01d6-408c-a6c8-fe61cb097aae-kube-api-access-qbqdz\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.345273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.344921    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-metrics-certs\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.345273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.344984    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f211138-01d6-408c-a6c8-fe61cb097aae-openshift-service-ca\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.346467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.346439    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f211138-01d6-408c-a6c8-fe61cb097aae-openshift-service-ca\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.360821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.360790    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-metrics-certs\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.375051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.375011    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbqdz\" (UniqueName: \"kubernetes.io/projected/5f211138-01d6-408c-a6c8-fe61cb097aae-kube-api-access-qbqdz\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.383166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.383125    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-maas-api-tls\") pod \"maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:17.467772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.467742    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:09:17.589674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.589634    5282 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:09:17.590005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.589965    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 20:09:17.699269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:17.698506    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:18.001231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:18.000045    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:18.347901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:18.344604    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"]
Nov 03 20:09:18.526556 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:09:18.526458    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f211138_01d6_408c_a6c8_fe61cb097aae.slice/crio-f1533effc440362d2c2ede1cdfdf67cef2994ed3dde6d6726cafb673e5a4e244 WatchSource:0}: Error finding container f1533effc440362d2c2ede1cdfdf67cef2994ed3dde6d6726cafb673e5a4e244: Status 404 returned error can't find the container with id f1533effc440362d2c2ede1cdfdf67cef2994ed3dde6d6726cafb673e5a4e244
Nov 03 20:09:18.599343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:18.599253    5282 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:09:18.599514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:18.599342    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 20:09:18.829518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:18.829485    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:19.115076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:19.115006    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k" event={"ID":"5f211138-01d6-408c-a6c8-fe61cb097aae","Type":"ContainerStarted","Data":"f1533effc440362d2c2ede1cdfdf67cef2994ed3dde6d6726cafb673e5a4e244"}
Nov 03 20:09:19.571097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:19.563884    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:19.594222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:19.594166    5282 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:09:19.594416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:19.594270    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 20:09:20.124584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.124548    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k" event={"ID":"5f211138-01d6-408c-a6c8-fe61cb097aae","Type":"ContainerStarted","Data":"9e8c8975b96452486fb50b5df621800f40c27b6e94589ffd3014735062c9301d"}
Nov 03 20:09:20.125050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.124633    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:20.146864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.146810    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k" podStartSLOduration=2.6083139109999998 podStartE2EDuration="3.146796417s" totalImagesPullingTime="538.482506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:20.143095406 +0000 UTC m=+1752.812296900" watchObservedRunningTime="2026-11-03 20:09:20.146796417 +0000 UTC m=+1752.815997886"
Nov 03 20:09:20.222834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.222793    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:20.587100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.587047    5282 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:09:20.587418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.587387    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 20:09:20.713450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.713416    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:20.950129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:20.950038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:21.581614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:21.581577    5282 patch_prober.go:29] interesting pod/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:09:21.582083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:21.581645    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 03 20:09:21.635960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:21.635927    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:22.391505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:22.391472    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:22.586421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:22.586394    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:09:22.587651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:22.587628    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:09:23.100161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:23.100130    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:23.203973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:23.203941    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:23.891796 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:23.891760    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:24.031233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:24.025591    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:24.664838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:24.664762    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:25.212233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:25.212165    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:25.386042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:25.386003    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:25.816515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:25.816479    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:26.098496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:26.098409    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:26.217906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:26.217859    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:09:26.266278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:26.266243    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:26.979441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:26.979411    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:27.655905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:27.655874    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:28.681819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:28.681780    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:29.560181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:29.560136    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:29.754530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:29.754490    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:29.886261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:29.886117    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:30.499335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:30.487322    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:30.608587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:30.608553    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:30.948646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:30.948539    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:30.970901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:30.970869    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"]
Nov 03 20:09:30.975520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:30.975500    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:30.987173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:30.987151    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1962f9-dockercfg-cqfmd\""
Nov 03 20:09:30.988697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:30.988674    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"]
Nov 03 20:09:31.107241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.089438    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30bd10e-d74d-4f45-b0b5-21ca277c459f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:31.107241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.089647    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj7c5\" (UniqueName: \"kubernetes.io/projected/f30bd10e-d74d-4f45-b0b5-21ca277c459f-kube-api-access-wj7c5\") pod \"payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:31.107241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.096508    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1962f9\""
Nov 03 20:09:31.115713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.108068    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:31.225302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.212160    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30bd10e-d74d-4f45-b0b5-21ca277c459f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:31.225810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.225781    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj7c5\" (UniqueName: \"kubernetes.io/projected/f30bd10e-d74d-4f45-b0b5-21ca277c459f-kube-api-access-wj7c5\") pod \"payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:31.282145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.282106    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30bd10e-d74d-4f45-b0b5-21ca277c459f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:31.283044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.283015    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj7c5\" (UniqueName: \"kubernetes.io/projected/f30bd10e-d74d-4f45-b0b5-21ca277c459f-kube-api-access-wj7c5\") pod \"payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:31.305789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.305490    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:31.515124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.515089    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:31.616283 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:09:31.616237    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:09:31.798133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:31.798036    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"]
Nov 03 20:09:32.453455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.453419    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"]
Nov 03 20:09:32.458255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.458232    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.482474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.482430    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b9dc48-openshift-default-dockercfg-wg8jd\""
Nov 03 20:09:32.489637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.489603    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"]
Nov 03 20:09:32.495615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.495582    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2" event={"ID":"f30bd10e-d74d-4f45-b0b5-21ca277c459f","Type":"ContainerStarted","Data":"4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f"}
Nov 03 20:09:32.495773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.495682    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:32.495773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.495710    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2" event={"ID":"f30bd10e-d74d-4f45-b0b5-21ca277c459f","Type":"ContainerStarted","Data":"ce6080655113d10426e4cd969a8f15a87e4f11dd55567f1f9235bedb01ae19c5"}
Nov 03 20:09:32.518717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.518667    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2" podStartSLOduration=2.518654657 podStartE2EDuration="2.518654657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:32.513515013 +0000 UTC m=+1765.182716498" watchObservedRunningTime="2026-11-03 20:09:32.518654657 +0000 UTC m=+1765.187856169"
Nov 03 20:09:32.614192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.614160    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.614380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.614235    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.614380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.614259    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.614500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.614477    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.614590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.614562    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.614734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.614715    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.614957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.614933    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27507f86-3733-48d7-9223-c5fb07e94f9e-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.615109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.615089    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.615185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.615167    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlj8\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-kube-api-access-mzlj8\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.729750    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.729862    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.729965    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.730146    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27507f86-3733-48d7-9223-c5fb07e94f9e-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.730271    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.730322    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlj8\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-kube-api-access-mzlj8\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.730413    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.730465    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.730685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.730519    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.734223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.731248    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.734223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.732946    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.734223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.734015    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.742225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.734664    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:32.758218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.753111    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.758218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.753329    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27507f86-3733-48d7-9223-c5fb07e94f9e-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.825221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.788581    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.825221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.789148    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.825221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.811805    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:32.839220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:32.829990    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlj8\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-kube-api-access-mzlj8\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:33.052287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.051608    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:33.101146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.101058    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:33.273741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.273691    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:33.479274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.456213    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:33.704832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.704436    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:33.712113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.710482    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"]
Nov 03 20:09:33.714215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.713089    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:33.714215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.713215    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:33.714215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:33.713269    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:34.290029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:34.289992    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:34.793223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:34.792470    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" event={"ID":"27507f86-3733-48d7-9223-c5fb07e94f9e","Type":"ContainerStarted","Data":"da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b"}
Nov 03 20:09:34.793223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:34.792542    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" event={"ID":"27507f86-3733-48d7-9223-c5fb07e94f9e","Type":"ContainerStarted","Data":"6695b9bac2c95805b81b49c18d7c4d315848e56de23aea3666debb7a9495b119"}
Nov 03 20:09:34.819377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:34.819326    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podStartSLOduration=2.819312681 podStartE2EDuration="2.819312681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:34.817310459 +0000 UTC m=+1767.486511931" watchObservedRunningTime="2026-11-03 20:09:34.819312681 +0000 UTC m=+1767.488514150"
Nov 03 20:09:35.092451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:35.092370    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:35.101906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:35.101883    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:35.107667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:35.107643    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:35.800928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:35.800903    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:35.801829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:35.801806    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:09:36.030390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:36.030359    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.031453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:37.031416    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.162103 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:37.162062    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.496222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:37.496163    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:37.642607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:37.640495    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:38.239013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:38.238966    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:38.566389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:38.562714    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:39.153222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.149159    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:39.376771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.374671    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:39.411481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.411389    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"]
Nov 03 20:09:39.418445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.418415    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:39.428437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.428407    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"]
Nov 03 20:09:39.645221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.609767    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klcrm\" (UniqueName: \"kubernetes.io/projected/0bccfe06-8c2d-44e2-aa95-921392f4b108-kube-api-access-klcrm\") pod \"payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:39.693829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.666906    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b9dc48\""
Nov 03 20:09:39.693829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.667246    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b9dc48-dockercfg-42kxp\""
Nov 03 20:09:39.693829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.674232    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:39.693829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.692309    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bccfe06-8c2d-44e2-aa95-921392f4b108-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:39.844234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.841175    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klcrm\" (UniqueName: \"kubernetes.io/projected/0bccfe06-8c2d-44e2-aa95-921392f4b108-kube-api-access-klcrm\") pod \"payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:39.844234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.841350    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bccfe06-8c2d-44e2-aa95-921392f4b108-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:39.844234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.842612    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bccfe06-8c2d-44e2-aa95-921392f4b108-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:39.883037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:39.882943    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klcrm\" (UniqueName: \"kubernetes.io/projected/0bccfe06-8c2d-44e2-aa95-921392f4b108-kube-api-access-klcrm\") pod \"payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:40.108102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:40.108062    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:40.340446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:40.340413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:40.589001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:40.588443    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"]
Nov 03 20:09:40.604416 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:09:40.604378    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bccfe06_8c2d_44e2_aa95_921392f4b108.slice/crio-d03f3abd5ac114773687f2014dc7e6e1b7c4eaa53e7c967b1ef54f9473ed8d88 WatchSource:0}: Error finding container d03f3abd5ac114773687f2014dc7e6e1b7c4eaa53e7c967b1ef54f9473ed8d88: Status 404 returned error can't find the container with id d03f3abd5ac114773687f2014dc7e6e1b7c4eaa53e7c967b1ef54f9473ed8d88
Nov 03 20:09:41.099822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:41.099778    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:41.341392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:41.341351    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w" event={"ID":"0bccfe06-8c2d-44e2-aa95-921392f4b108","Type":"ContainerStarted","Data":"7bf066e4780e2bb76172a1232f40f7d39f026a62c62666c22164b08c167faa87"}
Nov 03 20:09:41.341595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:41.341570    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:41.341644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:41.341609    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w" event={"ID":"0bccfe06-8c2d-44e2-aa95-921392f4b108","Type":"ContainerStarted","Data":"d03f3abd5ac114773687f2014dc7e6e1b7c4eaa53e7c967b1ef54f9473ed8d88"}
Nov 03 20:09:41.360788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:41.360698    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w" podStartSLOduration=2.36068621 podStartE2EDuration="2.36068621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:41.357576758 +0000 UTC m=+1774.026778231" watchObservedRunningTime="2026-11-03 20:09:41.36068621 +0000 UTC m=+1774.029887713"
Nov 03 20:09:41.971659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:41.971616    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:42.699652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:42.699618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:42.826266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:42.821855    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:43.607953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:43.607920    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:43.826277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:43.823418    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:09:44.399546 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:44.399501    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:45.099144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:45.099105    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:45.735315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:45.735270    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:46.363599 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:46.363564    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:47.197279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:47.197246    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:47.879110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:47.879079    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:48.478223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:48.478178    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:48.535076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:48.534480    5282 generic.go:356] "Generic (PLEG): container finished" podID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerID="773e386b3e898f7d23d056b40466205291ad5913f6ade00184935326b1639941" exitCode=0
Nov 03 20:09:48.535076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:48.534633    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" event={"ID":"2e13f182-06e1-4122-922b-ebc4ebeca7ce","Type":"ContainerDied","Data":"773e386b3e898f7d23d056b40466205291ad5913f6ade00184935326b1639941"}
Nov 03 20:09:48.536693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:48.535575    5282 scope.go:122] "RemoveContainer" containerID="773e386b3e898f7d23d056b40466205291ad5913f6ade00184935326b1639941"
Nov 03 20:09:48.540496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:48.539518    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:48.540496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:48.539652    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:48.540496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:48.539704    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:09:49.135803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:49.135707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:49.291448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:49.291408    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:49.544949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:49.544918    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" event={"ID":"2e13f182-06e1-4122-922b-ebc4ebeca7ce","Type":"ContainerStarted","Data":"ef9d0a2c57221dde360adfa62464de720d2da2f58f4c69b00ace66277c92aabd"}
Nov 03 20:09:49.679600 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:49.679569    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:49.683128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:49.683051    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 20:09:49.683402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:49.683370    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 20:09:49.849579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:49.841192    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:50.160294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:50.157704    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:50.676747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:50.676309    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 20:09:50.676747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:50.676366    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 20:09:50.903325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:50.903293    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:51.101777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:51.101740    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:51.714224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:51.713839    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 20:09:51.714224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:51.713915    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 20:09:51.732744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:51.732695    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:52.403710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:52.402398    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:09:52.504112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:52.504066    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:52.681897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:52.681804    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 20:09:52.682177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:52.682134    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 20:09:53.171594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:53.171563    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:53.537606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:53.537570    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:53.650419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:53.650380    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:53.694617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:53.694580    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 20:09:53.694913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:53.694882    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 20:09:53.908280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:53.907327    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:54.498222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:54.498165    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:54.680228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:54.679591    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 20:09:54.680228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:54.679670    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 20:09:55.094713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:55.094681    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:55.693285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:55.689646    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 20:09:55.693285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:55.689725    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 20:09:55.972285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:55.963593    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:56.680289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:56.680256    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:56.681288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:56.681272    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:09:56.809956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:56.809925    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:57.496937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:57.496898    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:58.093538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:58.093497    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:58.786217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:58.785840    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:59.348726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:59.348694    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:09:59.743349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:09:59.743307    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:00.102005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.101922    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:00.381621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.381542    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:00.745790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.745757    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"]
Nov 03 20:10:00.757493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.754632    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.757493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.755406    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"]
Nov 03 20:10:00.761241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.760726    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cd7f5c-openshift-default-dockercfg-dsl78\""
Nov 03 20:10:00.769271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.769240    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-credential-socket\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.769492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.769471    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istiod-ca-cert\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.770080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.770058    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-envoy\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.770297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.770279    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-podinfo\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.770515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.770497    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-data\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.770804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.770782    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-token\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.770951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.770934    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-socket\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.771256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.771237    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-certs\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.772001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.771978    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9dt\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-kube-api-access-4g9dt\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.802633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.802570    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:00.874179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874135    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-token\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874189    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-socket\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874256    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-certs\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874302    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9dt\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-kube-api-access-4g9dt\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874412    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-credential-socket\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874428    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istiod-ca-cert\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874529    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-envoy\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874635    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-podinfo\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874707    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-data\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874717    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-socket\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874742    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-certs\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874804    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-credential-socket\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.874980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.874952    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-data\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.875144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.875127    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istiod-ca-cert\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.876911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.876882    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-envoy\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.877068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.877053    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-podinfo\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.885252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.885223    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-token\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:00.885352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:00.885307    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9dt\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-kube-api-access-4g9dt\") pod \"e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") " pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:01.085281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:01.079432    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:01.085281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:01.082144    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:01.618837 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:10:01.618741    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:10:01.691499 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:01.691470    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bb8111_a2fd_4e84_a7da_e63466d8b6d9.slice/crio-aa58c9449a1bcd981e2a253f3b70a57f497809f1fff85354fea7a915a6aaf0f5 WatchSource:0}: Error finding container aa58c9449a1bcd981e2a253f3b70a57f497809f1fff85354fea7a915a6aaf0f5: Status 404 returned error can't find the container with id aa58c9449a1bcd981e2a253f3b70a57f497809f1fff85354fea7a915a6aaf0f5
Nov 03 20:10:01.691673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:01.691649    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"]
Nov 03 20:10:01.694134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:01.694098    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:01.694253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:01.694166    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:01.694253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:01.694216    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:01.720372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:01.720340    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:02.257552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:02.257517    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" event={"ID":"71bb8111-a2fd-4e84-a7da-e63466d8b6d9","Type":"ContainerStarted","Data":"124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4"}
Nov 03 20:10:02.257785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:02.257580    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" event={"ID":"71bb8111-a2fd-4e84-a7da-e63466d8b6d9","Type":"ContainerStarted","Data":"aa58c9449a1bcd981e2a253f3b70a57f497809f1fff85354fea7a915a6aaf0f5"}
Nov 03 20:10:02.277931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:02.277868    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podStartSLOduration=2.277846878 podStartE2EDuration="2.277846878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:02.275723029 +0000 UTC m=+1794.944924509" watchObservedRunningTime="2026-11-03 20:10:02.277846878 +0000 UTC m=+1794.947048350"
Nov 03 20:10:02.713668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:02.708934    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:02.830242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:02.826863    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:03.089446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:03.089416    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:03.095252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:03.095189    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:03.265552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:03.265519    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:03.266416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:03.266397    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:10:03.717137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:03.717099    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:04.884538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:04.884505    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:05.094219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:05.091227    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:06.121633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.121598    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:06.206112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.206076    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"]
Nov 03 20:10:06.211008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.210989    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.218579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.218556    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cd7f5c\""
Nov 03 20:10:06.218919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.218895    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cd7f5c-dockercfg-cmvsz\""
Nov 03 20:10:06.221926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.221796    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"]
Nov 03 20:10:06.236233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.236183    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"]
Nov 03 20:10:06.241544 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.241523    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:06.254152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.254126    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"]
Nov 03 20:10:06.255665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.255545    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b67ss\" (UniqueName: \"kubernetes.io/projected/21072de7-af50-4445-bc9b-39f704e02502-kube-api-access-b67ss\") pod \"payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.256603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.255721    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21072de7-af50-4445-bc9b-39f704e02502-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.270341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.270317    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:06.424289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.398847    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b734719e-2eb8-40c7-8339-47831a5ce3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:06.493375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.493312    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b67ss\" (UniqueName: \"kubernetes.io/projected/21072de7-af50-4445-bc9b-39f704e02502-kube-api-access-b67ss\") pod \"payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.515260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.493672    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21072de7-af50-4445-bc9b-39f704e02502-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.602460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.602419    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqtg\" (UniqueName: \"kubernetes.io/projected/b734719e-2eb8-40c7-8339-47831a5ce3ad-kube-api-access-dwqtg\") pod \"payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:06.639216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.502131    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21072de7-af50-4445-bc9b-39f704e02502-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.687243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.687137    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b67ss\" (UniqueName: \"kubernetes.io/projected/21072de7-af50-4445-bc9b-39f704e02502-kube-api-access-b67ss\") pod \"payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") " pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.757219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.753214    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b734719e-2eb8-40c7-8339-47831a5ce3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:06.757219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.753389    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqtg\" (UniqueName: \"kubernetes.io/projected/b734719e-2eb8-40c7-8339-47831a5ce3ad-kube-api-access-dwqtg\") pod \"payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:06.757219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.754946    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b734719e-2eb8-40c7-8339-47831a5ce3ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:06.787236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.786538    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqtg\" (UniqueName: \"kubernetes.io/projected/b734719e-2eb8-40c7-8339-47831a5ce3ad-kube-api-access-dwqtg\") pod \"payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:06.837491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.837458    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:06.863882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:06.863760    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:07.122299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.122268    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"]
Nov 03 20:10:07.122713 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:07.122465    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21072de7_af50_4445_bc9b_39f704e02502.slice/crio-ca75aeeb05927a79fedca8218abef4374f2c5181fbae308176196e7ff6bb406f WatchSource:0}: Error finding container ca75aeeb05927a79fedca8218abef4374f2c5181fbae308176196e7ff6bb406f: Status 404 returned error can't find the container with id ca75aeeb05927a79fedca8218abef4374f2c5181fbae308176196e7ff6bb406f
Nov 03 20:10:07.162060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.162036    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"]
Nov 03 20:10:07.164763 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:07.164738    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb734719e_2eb8_40c7_8339_47831a5ce3ad.slice/crio-d34bf394caa8431183a171999f03adac63cedf98d5d8c5b3c291e264ff88c968 WatchSource:0}: Error finding container d34bf394caa8431183a171999f03adac63cedf98d5d8c5b3c291e264ff88c968: Status 404 returned error can't find the container with id d34bf394caa8431183a171999f03adac63cedf98d5d8c5b3c291e264ff88c968
Nov 03 20:10:07.167622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.167593    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:07.283599 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.281274    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:07.623661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.623618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:07.642170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.642119    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr" event={"ID":"b734719e-2eb8-40c7-8339-47831a5ce3ad","Type":"ContainerStarted","Data":"44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be"}
Nov 03 20:10:07.642413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.642248    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr" event={"ID":"b734719e-2eb8-40c7-8339-47831a5ce3ad","Type":"ContainerStarted","Data":"d34bf394caa8431183a171999f03adac63cedf98d5d8c5b3c291e264ff88c968"}
Nov 03 20:10:07.643156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.643118    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:07.644807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.644766    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd" event={"ID":"21072de7-af50-4445-bc9b-39f704e02502","Type":"ContainerStarted","Data":"91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6"}
Nov 03 20:10:07.644936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.644813    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd" event={"ID":"21072de7-af50-4445-bc9b-39f704e02502","Type":"ContainerStarted","Data":"ca75aeeb05927a79fedca8218abef4374f2c5181fbae308176196e7ff6bb406f"}
Nov 03 20:10:07.645072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.645055    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:07.664737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.664687    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr" podStartSLOduration=1.664674222 podStartE2EDuration="1.664674222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:07.659211632 +0000 UTC m=+1800.328413101" watchObservedRunningTime="2026-11-03 20:10:07.664674222 +0000 UTC m=+1800.333875691"
Nov 03 20:10:07.678479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.678438    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd" podStartSLOduration=1.678427021 podStartE2EDuration="1.678427021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:07.674916535 +0000 UTC m=+1800.344118007" watchObservedRunningTime="2026-11-03 20:10:07.678427021 +0000 UTC m=+1800.347628491"
Nov 03 20:10:07.732682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.732645    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:07.875579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.871746    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:07.880221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:07.878685    5282 scope.go:122] "RemoveContainer" containerID="b05588c98e2e20f6baf1af24af7f84c2230c91e1d1a5568b965ac83cb6408431"
Nov 03 20:10:08.109255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:08.108997    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:08.640951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:08.640916    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:08.922172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:08.922097    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:09.996453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:09.996415    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:10.696103 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:10.696067    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:11.588425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:11.588389    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:12.002127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:12.002094    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:12.129583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:12.129540    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:12.522679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:12.522637    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:10:12.526258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:12.526215    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:13.414520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.414487    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:13.460586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.460549    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"]
Nov 03 20:10:13.467530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.467502    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:13.468852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.468824    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"]
Nov 03 20:10:13.471998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.471973    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a476f8\""
Nov 03 20:10:13.472728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.472706    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a476f8-dockercfg-sx7cs\""
Nov 03 20:10:13.564950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.564918    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:13.596173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.596142    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrn8w\" (UniqueName: \"kubernetes.io/projected/68cfe518-7312-4bbb-8700-f8a165a77251-kube-api-access-nrn8w\") pod \"payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:13.596436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.596414    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cfe518-7312-4bbb-8700-f8a165a77251-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:13.703977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.703881    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrn8w\" (UniqueName: \"kubernetes.io/projected/68cfe518-7312-4bbb-8700-f8a165a77251-kube-api-access-nrn8w\") pod \"payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:13.704453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.704422    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cfe518-7312-4bbb-8700-f8a165a77251-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:13.705883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.705859    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cfe518-7312-4bbb-8700-f8a165a77251-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:13.769001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.768966    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrn8w\" (UniqueName: \"kubernetes.io/projected/68cfe518-7312-4bbb-8700-f8a165a77251-kube-api-access-nrn8w\") pod \"payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:13.799221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:13.797539    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:14.001130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.001065    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:14.276284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.275819    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:14.460723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.460099    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"]
Nov 03 20:10:14.485930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.485758    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.493239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.493187    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5b918c5d-openshift-default-dockercfg-q9rch\""
Nov 03 20:10:14.494189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.494160    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"]
Nov 03 20:10:14.563459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.563375    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:14.582330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.582291    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-token\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.582491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.582358    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c004d68-fd7d-4827-ab19-fed3800b9e07-istiod-ca-cert\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.582545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.582503    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-socket\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.582624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.582608    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-credential-socket\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.582798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.582785    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-data\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.582860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.582843    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-certs\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.582930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.582906    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sqws\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-kube-api-access-7sqws\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.583076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.583058    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-envoy\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.583131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.583120    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-podinfo\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.599457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.599429    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"]
Nov 03 20:10:14.599736 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:14.599715    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cfe518_7312_4bbb_8700_f8a165a77251.slice/crio-9e063c7538673dd2f0d8940aab01b3e3ab28f1fb12e923d8bcd6d07709f8c56a WatchSource:0}: Error finding container 9e063c7538673dd2f0d8940aab01b3e3ab28f1fb12e923d8bcd6d07709f8c56a: Status 404 returned error can't find the container with id 9e063c7538673dd2f0d8940aab01b3e3ab28f1fb12e923d8bcd6d07709f8c56a
Nov 03 20:10:14.719224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.718611    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-token\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.718686    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c004d68-fd7d-4827-ab19-fed3800b9e07-istiod-ca-cert\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.718759    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-socket\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.718843    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-credential-socket\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.719073    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-data\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.719161    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-certs\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.719188    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sqws\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-kube-api-access-7sqws\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.719302    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-envoy\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.719754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.719353    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-podinfo\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.742489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.742303    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-credential-socket\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.751232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.743303    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c004d68-fd7d-4827-ab19-fed3800b9e07-istiod-ca-cert\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.751232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.743553    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-socket\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.751232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.743770    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-certs\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.751232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.743985    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-data\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.785015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.784975    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-envoy\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.787370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.787309    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-podinfo\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.847276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.805084    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sqws\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-kube-api-access-7sqws\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.847276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.824820    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-token\") pod \"e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") " pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:14.853595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.853557    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:14.958236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:14.956766    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd" event={"ID":"68cfe518-7312-4bbb-8700-f8a165a77251","Type":"ContainerStarted","Data":"9e063c7538673dd2f0d8940aab01b3e3ab28f1fb12e923d8bcd6d07709f8c56a"}
Nov 03 20:10:15.116315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.115390    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:15.258254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.258212    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:15.467729 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:15.467694    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c004d68_fd7d_4827_ab19_fed3800b9e07.slice/crio-9c17c6e12f2e2f751d5a096076a36282d201949e762065ceab35d9b103d63622 WatchSource:0}: Error finding container 9c17c6e12f2e2f751d5a096076a36282d201949e762065ceab35d9b103d63622: Status 404 returned error can't find the container with id 9c17c6e12f2e2f751d5a096076a36282d201949e762065ceab35d9b103d63622
Nov 03 20:10:15.469324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.469298    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"]
Nov 03 20:10:15.470581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.470549    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:15.470641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.470611    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:15.470674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.470638    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:15.967317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.967282    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd" event={"ID":"68cfe518-7312-4bbb-8700-f8a165a77251","Type":"ContainerStarted","Data":"c1944a589cede9a8524a14036b574e5ee655400e60cb4454341c78e27e9523ba"}
Nov 03 20:10:15.967492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.967439    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:15.968946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.968924    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" event={"ID":"3c004d68-fd7d-4827-ab19-fed3800b9e07","Type":"ContainerStarted","Data":"523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a"}
Nov 03 20:10:15.969042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.968960    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" event={"ID":"3c004d68-fd7d-4827-ab19-fed3800b9e07","Type":"ContainerStarted","Data":"9c17c6e12f2e2f751d5a096076a36282d201949e762065ceab35d9b103d63622"}
Nov 03 20:10:15.987550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:15.987507    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd" podStartSLOduration=2.98749505 podStartE2EDuration="2.98749505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:15.982698031 +0000 UTC m=+1808.651899502" watchObservedRunningTime="2026-11-03 20:10:15.98749505 +0000 UTC m=+1808.656696520"
Nov 03 20:10:16.006404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:16.006352    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" podStartSLOduration=2.006338863 podStartE2EDuration="2.006338863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:16.001182441 +0000 UTC m=+1808.670383939" watchObservedRunningTime="2026-11-03 20:10:16.006338863 +0000 UTC m=+1808.675540332"
Nov 03 20:10:16.139225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:16.133817    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:16.215641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:16.209870    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:16.419446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:16.419413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:16.977286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:16.977248    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:16.978112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:16.978092    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:10:17.533608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:17.533551    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:18.321071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:18.321032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:18.441250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:18.441188    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:18.624588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:18.624499    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:18.721219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:18.719436    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:10:19.565134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:19.565011    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:19.671749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:19.671722    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:10:19.680941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:19.680915    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:19.799084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:19.799052    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:20.322234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.322161    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:20.385385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.384649    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"]
Nov 03 20:10:20.438870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.438836    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"]
Nov 03 20:10:20.439123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.439107    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"]
Nov 03 20:10:20.440776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.440743    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:20.442372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.442346    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:20.450611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.450572    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5b918c5d\""
Nov 03 20:10:20.504253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.504044    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5b918c5d-dockercfg-7p9ph\""
Nov 03 20:10:20.609272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.599078    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f98f1f-2e73-466d-9ba7-9725790a772a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:20.629236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.617338    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cvv\" (UniqueName: \"kubernetes.io/projected/86f98f1f-2e73-466d-9ba7-9725790a772a-kube-api-access-48cvv\") pod \"payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:20.694759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.694712    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"]
Nov 03 20:10:20.729250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.700357    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:20.809311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.785621    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cvv\" (UniqueName: \"kubernetes.io/projected/86f98f1f-2e73-466d-9ba7-9725790a772a-kube-api-access-48cvv\") pod \"payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:20.809311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.785824    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f98f1f-2e73-466d-9ba7-9725790a772a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:20.809311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:20.787002    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f98f1f-2e73-466d-9ba7-9725790a772a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:21.011508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.005123    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:21.013628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.013592    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") " pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:21.013767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.013711    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjhs\" (UniqueName: \"kubernetes.io/projected/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-kube-api-access-gjjhs\") pod \"payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") " pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:21.021130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.021104    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cvv\" (UniqueName: \"kubernetes.io/projected/86f98f1f-2e73-466d-9ba7-9725790a772a-kube-api-access-48cvv\") pod \"payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:21.116936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.116898    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") " pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:21.117149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.116994    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjhs\" (UniqueName: \"kubernetes.io/projected/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-kube-api-access-gjjhs\") pod \"payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") " pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:21.117700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.117678    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-plugins-config-volume\") pod \"payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") " pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:21.125162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.125136    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjhs\" (UniqueName: \"kubernetes.io/projected/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-kube-api-access-gjjhs\") pod \"payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") " pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:21.309459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.309420    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:21.317977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.317953    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:21.549472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.549439    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:21.714006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.707459    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"]
Nov 03 20:10:21.719267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:21.719111    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"]
Nov 03 20:10:21.720660 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:21.720618    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f98f1f_2e73_466d_9ba7_9725790a772a.slice/crio-00645b4e49b7a96c7383bbc91690316936131c1a62b04779ad9f88dfe9010c01 WatchSource:0}: Error finding container 00645b4e49b7a96c7383bbc91690316936131c1a62b04779ad9f88dfe9010c01: Status 404 returned error can't find the container with id 00645b4e49b7a96c7383bbc91690316936131c1a62b04779ad9f88dfe9010c01
Nov 03 20:10:22.068433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.068391    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx" event={"ID":"86f98f1f-2e73-466d-9ba7-9725790a772a","Type":"ContainerStarted","Data":"ac52ed2ec8cbff32573d84e063d090547e01266f5abf1fa50f40842d0143229f"}
Nov 03 20:10:22.068628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.068453    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx" event={"ID":"86f98f1f-2e73-466d-9ba7-9725790a772a","Type":"ContainerStarted","Data":"00645b4e49b7a96c7383bbc91690316936131c1a62b04779ad9f88dfe9010c01"}
Nov 03 20:10:22.068628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.068481    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:22.070357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.070330    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8" event={"ID":"13c11ce5-b830-4da0-87fe-0f57fc49aeb2","Type":"ContainerStarted","Data":"5c3aad9ad5d0586d7f9bff5cd7a695b9358642e4215407f12eed2016562be9c0"}
Nov 03 20:10:22.070475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.070377    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8" event={"ID":"13c11ce5-b830-4da0-87fe-0f57fc49aeb2","Type":"ContainerStarted","Data":"ff2b451e419e530a22ee6e5440f2629eab4938ef9c19a11a193e1c5b0ee8e4a8"}
Nov 03 20:10:22.070475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.070409    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:22.091365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.091313    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx" podStartSLOduration=2.0913002 podStartE2EDuration="2.0913002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:22.087914415 +0000 UTC m=+1814.757115885" watchObservedRunningTime="2026-11-03 20:10:22.0913002 +0000 UTC m=+1814.760501670"
Nov 03 20:10:22.105136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.105089    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8" podStartSLOduration=2.105077631 podStartE2EDuration="2.105077631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:22.102568345 +0000 UTC m=+1814.771769815" watchObservedRunningTime="2026-11-03 20:10:22.105077631 +0000 UTC m=+1814.774279100"
Nov 03 20:10:22.420287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.419826    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:22.557051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:22.556956    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:23.285031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:23.284992    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:24.066084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:24.066053    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:25.728307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:25.728266    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:26.421777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:26.421744    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:26.979972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:26.979947    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:10:27.116745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:27.116712    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:27.898167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:27.898138    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:28.185932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:28.185856    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:28.319235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:28.319182    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:28.453342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:28.453257    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:28.853597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:28.848696    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:29.455793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:29.455756    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:29.629947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:29.629910    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:29.937498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:29.937462    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:30.186260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:30.164412    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:31.397677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:31.396251    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:31.678521 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:10:31.678433    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:10:32.506547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:32.506465    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:33.169352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.169321    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:10:33.169749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.169729    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:10:33.298492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.298460    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"]
Nov 03 20:10:33.303458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.303439    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.321432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.321402    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cad531-openshift-default-dockercfg-gj4fh\""
Nov 03 20:10:33.348543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.348512    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"]
Nov 03 20:10:33.358081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.358053    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"]
Nov 03 20:10:33.364319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.364294    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.371324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.371297    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"]
Nov 03 20:10:33.384866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.384834    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.384919    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.384954    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385004    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385044    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4n89\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-kube-api-access-v4n89\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385088    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385187    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385248    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385264    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8918da4-955f-455e-bd6d-a5f7d58a2867-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385336    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385384    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385453    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385511    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nckwx\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-kube-api-access-nckwx\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385561    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.385756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385603    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385627    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385655    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.385756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.385689    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.487312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487283    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.487469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487391    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.487570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487549    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.487612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487583    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8918da4-955f-455e-bd6d-a5f7d58a2867-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.487676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487660    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.487768 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487732    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.487768 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487748    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.487891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487829    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.487955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487899    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nckwx\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-kube-api-access-nckwx\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.487955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487930    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487957    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488004    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488024    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.487958    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488028    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488087    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488128    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488155    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488262    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488285    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488320    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488371    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8918da4-955f-455e-bd6d-a5f7d58a2867-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488383    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488396    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488466    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488491    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.488799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488545    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4n89\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-kube-api-access-v4n89\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.488799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.488659    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.490632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.490601    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.490890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.490865    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.490979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.490925    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.491137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.491118    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.496062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.496036    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.496062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.496051    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4n89\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-kube-api-access-v4n89\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.496517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.496501    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.499400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.499380    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nckwx\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-kube-api-access-nckwx\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.502345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.502323    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:33.660181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.658574    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:33.685257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.684552    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:33.994827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.994799    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"]
Nov 03 20:10:33.994912 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:33.994862    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8918da4_955f_455e_bd6d_a5f7d58a2867.slice/crio-ad488acda0beb76e0b282cd1563142bedce88153dfc21c88e7cd93f191af19d6 WatchSource:0}: Error finding container ad488acda0beb76e0b282cd1563142bedce88153dfc21c88e7cd93f191af19d6: Status 404 returned error can't find the container with id ad488acda0beb76e0b282cd1563142bedce88153dfc21c88e7cd93f191af19d6
Nov 03 20:10:33.997560 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:10:33.997535    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4a2e6f_3a4c_4b2a_ba5b_4fafdf736cce.slice/crio-94b929caa4e5320f0632bd8787dddef46b5b411509d16170537d881fc14ca571 WatchSource:0}: Error finding container 94b929caa4e5320f0632bd8787dddef46b5b411509d16170537d881fc14ca571: Status 404 returned error can't find the container with id 94b929caa4e5320f0632bd8787dddef46b5b411509d16170537d881fc14ca571
Nov 03 20:10:33.997948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.997907    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:33.998028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.997997    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:33.998089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:33.998053    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:34.000168 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.000148    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"]
Nov 03 20:10:34.000979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.000947    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:34.001045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.001024    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:34.001085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.001073    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:10:34.324874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.324790    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:34.633979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.633878    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" event={"ID":"a8918da4-955f-455e-bd6d-a5f7d58a2867","Type":"ContainerStarted","Data":"aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16"}
Nov 03 20:10:34.633979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.633943    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" event={"ID":"a8918da4-955f-455e-bd6d-a5f7d58a2867","Type":"ContainerStarted","Data":"ad488acda0beb76e0b282cd1563142bedce88153dfc21c88e7cd93f191af19d6"}
Nov 03 20:10:34.635991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.635956    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" event={"ID":"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce","Type":"ContainerStarted","Data":"097866ceb1f054b3bbc8921db1ee70982b95e515f27fca67463f0b326e1f3cce"}
Nov 03 20:10:34.636138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.636007    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" event={"ID":"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce","Type":"ContainerStarted","Data":"94b929caa4e5320f0632bd8787dddef46b5b411509d16170537d881fc14ca571"}
Nov 03 20:10:34.659461 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.659414    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:34.662768 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.661371    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podStartSLOduration=1.6613530509999999 podStartE2EDuration="1.661353051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:34.659069785 +0000 UTC m=+1827.328271256" watchObservedRunningTime="2026-11-03 20:10:34.661353051 +0000 UTC m=+1827.330554521"
Nov 03 20:10:34.685452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.685417    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:34.712224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:34.706449    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podStartSLOduration=1.706425529 podStartE2EDuration="1.706425529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:34.684015202 +0000 UTC m=+1827.353216703" watchObservedRunningTime="2026-11-03 20:10:34.706425529 +0000 UTC m=+1827.375627000"
Nov 03 20:10:35.418223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:35.400850    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:35.689304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:35.683569    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:10:35.689304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:35.683630    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 20:10:35.700221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:35.699065    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:10:35.700221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:35.699124    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 20:10:36.261493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.261462    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"]
Nov 03 20:10:36.267590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.267570    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.321254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.297588    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5b918c5d-serving-cert\""
Nov 03 20:10:36.343601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.342728    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:36.348747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.348714    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"]
Nov 03 20:10:36.468133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.457754    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:36.514287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.511831    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-maas-api-tls\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.514287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.511989    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a3038a-192a-4c85-997a-daf9a0734b38-openshift-service-ca\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.514287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.512218    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-metrics-certs\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.514287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.512459    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2z7\" (UniqueName: \"kubernetes.io/projected/c3a3038a-192a-4c85-997a-daf9a0734b38-kube-api-access-wh2z7\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.668031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.667988    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:10:36.668231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.668052    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 20:10:36.745272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.721097    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:10:36.745272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.721154    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 20:10:36.745272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.740774    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-metrics-certs\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.745272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.740930    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2z7\" (UniqueName: \"kubernetes.io/projected/c3a3038a-192a-4c85-997a-daf9a0734b38-kube-api-access-wh2z7\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.745272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.741213    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-maas-api-tls\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.745272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.741368    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a3038a-192a-4c85-997a-daf9a0734b38-openshift-service-ca\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.753230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.747187    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-metrics-certs\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.753230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.750535    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a3038a-192a-4c85-997a-daf9a0734b38-openshift-service-ca\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.753230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.750852    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:36.755744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.755667    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-maas-api-tls\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.763319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.763290    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2z7\" (UniqueName: \"kubernetes.io/projected/c3a3038a-192a-4c85-997a-daf9a0734b38-kube-api-access-wh2z7\") pod \"maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:36.970219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:36.965535    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:37.435831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:37.435796    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:37.817220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:37.816546    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:10:37.817220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:37.816611    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 20:10:37.825791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:37.818632    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:10:37.825791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:37.818683    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 20:10:38.203342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:38.203302    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"]
Nov 03 20:10:38.345626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:38.345596    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:38.660322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:38.660274    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:10:38.660535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:38.660348    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 20:10:38.697221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:38.696443    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:10:38.697221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:38.696525    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 20:10:39.228116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.228080    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv" event={"ID":"c3a3038a-192a-4c85-997a-daf9a0734b38","Type":"ContainerStarted","Data":"3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa"}
Nov 03 20:10:39.228627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.228138    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv" event={"ID":"c3a3038a-192a-4c85-997a-daf9a0734b38","Type":"ContainerStarted","Data":"1c15cc16988027b856d39386fa36101da04294d4f6c26c53b70f141fb6a0d288"}
Nov 03 20:10:39.228627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.228379    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:39.250180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.250122    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv" podStartSLOduration=2.469332505 podStartE2EDuration="3.250105748s" totalImagesPullingTime="780.773243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:39.249523966 +0000 UTC m=+1831.918725438" watchObservedRunningTime="2026-11-03 20:10:39.250105748 +0000 UTC m=+1831.919307218"
Nov 03 20:10:39.305231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.304773    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:39.660088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.660003    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:10:39.660088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.660057    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 20:10:39.696221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.685513    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:10:39.696221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:39.685575    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 20:10:40.126438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.126288    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:40.447573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.447301    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"]
Nov 03 20:10:40.452229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.452171    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"]
Nov 03 20:10:40.459179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.459151    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:40.622221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.601265    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:40.710311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.710229    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:10:40.711227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.711170    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 20:10:40.731376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.731341    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:10:40.731691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:40.731663    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 20:10:41.403146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:41.403107    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:41.598221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:41.597890    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:41.659445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:41.659341    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 20:10:41.659445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:41.659415    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 20:10:41.685397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:41.685360    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 20:10:41.685565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:41.685411    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 20:10:42.665286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:42.665256    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:42.665729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:42.665674    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:42.666250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:42.666190    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:10:42.670076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:42.670041    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:42.724363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:42.723663    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:42.741220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:42.725515    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:42.741220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:42.736457    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:10:43.569244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:43.569188    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:44.325223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:44.319893    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:45.242021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.241985    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:45.447725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.447679    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk" podUID="c8bddb75-dbbc-4907-8c40-0ca54c150a92" containerName="payload-processing" containerID="cri-o://24d24fca98d790832eea4f7c978ed10291b9e91fba5380f364596c2f56d4c9c6" gracePeriod=25
Nov 03 20:10:45.452655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.452619    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx" podUID="b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" containerName="payload-pre-processing" containerID="cri-o://7a571410d3c100d550b049d1d7ba829233f1f98060bc8cc447bae4bc1a485aa9" gracePeriod=25
Nov 03 20:10:45.467448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.467423    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:10:45.503537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.503467    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:45.764436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.764350    5282 generic.go:356] "Generic (PLEG): container finished" podID="c8bddb75-dbbc-4907-8c40-0ca54c150a92" containerID="24d24fca98d790832eea4f7c978ed10291b9e91fba5380f364596c2f56d4c9c6" exitCode=0
Nov 03 20:10:45.764828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.764807    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk" event={"ID":"c8bddb75-dbbc-4907-8c40-0ca54c150a92","Type":"ContainerDied","Data":"24d24fca98d790832eea4f7c978ed10291b9e91fba5380f364596c2f56d4c9c6"}
Nov 03 20:10:45.823812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.823766    5282 generic.go:356] "Generic (PLEG): container finished" podID="b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" containerID="7a571410d3c100d550b049d1d7ba829233f1f98060bc8cc447bae4bc1a485aa9" exitCode=0
Nov 03 20:10:45.824010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.823858    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx" event={"ID":"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338","Type":"ContainerDied","Data":"7a571410d3c100d550b049d1d7ba829233f1f98060bc8cc447bae4bc1a485aa9"}
Nov 03 20:10:45.910602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.910574    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:10:45.938805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.938764    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"]
Nov 03 20:10:45.949921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:45.948758    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k" podUID="5f211138-01d6-408c-a6c8-fe61cb097aae" containerName="maas-api" containerID="cri-o://9e8c8975b96452486fb50b5df621800f40c27b6e94589ffd3014735062c9301d" gracePeriod=30
Nov 03 20:10:46.034117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.027516    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:46.048657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.048620    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8bddb75-dbbc-4907-8c40-0ca54c150a92-kube-api-access-x2hrb\" (UniqueName: \"kubernetes.io/projected/c8bddb75-dbbc-4907-8c40-0ca54c150a92-kube-api-access-x2hrb\") pod \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") "
Nov 03 20:10:46.048883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.048871    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8bddb75-dbbc-4907-8c40-0ca54c150a92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bddb75-dbbc-4907-8c40-0ca54c150a92-plugins-config-volume\") pod \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\" (UID: \"c8bddb75-dbbc-4907-8c40-0ca54c150a92\") "
Nov 03 20:10:46.049269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.049248    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8bddb75-dbbc-4907-8c40-0ca54c150a92-plugins-config-volume" pod "c8bddb75-dbbc-4907-8c40-0ca54c150a92" (UID: "c8bddb75-dbbc-4907-8c40-0ca54c150a92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:46.049479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.049462    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bddb75-dbbc-4907-8c40-0ca54c150a92-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:46.050785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.050767    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bddb75-dbbc-4907-8c40-0ca54c150a92-kube-api-access-x2hrb" pod "c8bddb75-dbbc-4907-8c40-0ca54c150a92" (UID: "c8bddb75-dbbc-4907-8c40-0ca54c150a92"). InnerVolumeSpecName "kube-api-access-x2hrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:46.065003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.064977    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:10:46.202222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.201825    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k" podUID="5f211138-01d6-408c-a6c8-fe61cb097aae" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.105:8443/health\": dial tcp 10.133.0.105:8443: connect: connection refused"
Nov 03 20:10:46.213224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.211762    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hrb\" (UniqueName: \"kubernetes.io/projected/c8bddb75-dbbc-4907-8c40-0ca54c150a92-kube-api-access-x2hrb\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:46.267223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.266897    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:46.458387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.456464    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:46.567744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.542145    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-plugins-config-volume\") pod \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") "
Nov 03 20:10:46.567744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.542468    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-kube-api-access-xxsh8\" (UniqueName: \"kubernetes.io/projected/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-kube-api-access-xxsh8\") pod \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\" (UID: \"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338\") "
Nov 03 20:10:46.567744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.559537    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-plugins-config-volume" pod "b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" (UID: "b76cd7cc-1bc1-401d-8a81-c1dd02e6c338"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:46.598453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.598405    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-kube-api-access-xxsh8" pod "b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" (UID: "b76cd7cc-1bc1-401d-8a81-c1dd02e6c338"). InnerVolumeSpecName "kube-api-access-xxsh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:46.657969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.657922    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxsh8\" (UniqueName: \"kubernetes.io/projected/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-kube-api-access-xxsh8\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:46.658309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.658269    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:46.725391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.725282    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:46.983271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.966035    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:46.992377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.990412    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"
Nov 03 20:10:46.992377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.992084    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx" event={"ID":"b76cd7cc-1bc1-401d-8a81-c1dd02e6c338","Type":"ContainerDied","Data":"fb287f3e22e0185c6b59a52ae284fd0239c17c963c327958361c2d5bccdbdb8c"}
Nov 03 20:10:46.992377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:46.992151    5282 scope.go:122] "RemoveContainer" containerID="7a571410d3c100d550b049d1d7ba829233f1f98060bc8cc447bae4bc1a485aa9"
Nov 03 20:10:47.010290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.009645    5282 generic.go:356] "Generic (PLEG): container finished" podID="5f211138-01d6-408c-a6c8-fe61cb097aae" containerID="9e8c8975b96452486fb50b5df621800f40c27b6e94589ffd3014735062c9301d" exitCode=0
Nov 03 20:10:47.010290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.009836    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k" event={"ID":"5f211138-01d6-408c-a6c8-fe61cb097aae","Type":"ContainerDied","Data":"9e8c8975b96452486fb50b5df621800f40c27b6e94589ffd3014735062c9301d"}
Nov 03 20:10:47.014244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.013652    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk" event={"ID":"c8bddb75-dbbc-4907-8c40-0ca54c150a92","Type":"ContainerDied","Data":"977a2ba78ba9e6843cbc23563572f5a5c49ecd127c466d5a38f24b98c3521d27"}
Nov 03 20:10:47.014244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.013753    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"
Nov 03 20:10:47.026685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.026654    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:10:47.026850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.026684    5282 scope.go:122] "RemoveContainer" containerID="24d24fca98d790832eea4f7c978ed10291b9e91fba5380f364596c2f56d4c9c6"
Nov 03 20:10:47.046109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.045865    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"]
Nov 03 20:10:47.050283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.050261    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-acd98d21-555b65457f-vcszx"]
Nov 03 20:10:47.065154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.065132    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"]
Nov 03 20:10:47.083303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.083281    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-gvnhk"]
Nov 03 20:10:47.205077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.205044    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f211138-01d6-408c-a6c8-fe61cb097aae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f211138-01d6-408c-a6c8-fe61cb097aae-openshift-service-ca\") pod \"5f211138-01d6-408c-a6c8-fe61cb097aae\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") "
Nov 03 20:10:47.205283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.205221    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f211138-01d6-408c-a6c8-fe61cb097aae-kube-api-access-qbqdz\" (UniqueName: \"kubernetes.io/projected/5f211138-01d6-408c-a6c8-fe61cb097aae-kube-api-access-qbqdz\") pod \"5f211138-01d6-408c-a6c8-fe61cb097aae\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") "
Nov 03 20:10:47.205283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.205258    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-maas-api-tls\") pod \"5f211138-01d6-408c-a6c8-fe61cb097aae\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") "
Nov 03 20:10:47.205387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.205365    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f211138-01d6-408c-a6c8-fe61cb097aae-openshift-service-ca" pod "5f211138-01d6-408c-a6c8-fe61cb097aae" (UID: "5f211138-01d6-408c-a6c8-fe61cb097aae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:47.205620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.205599    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-metrics-certs\") pod \"5f211138-01d6-408c-a6c8-fe61cb097aae\" (UID: \"5f211138-01d6-408c-a6c8-fe61cb097aae\") "
Nov 03 20:10:47.206265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.206244    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f211138-01d6-408c-a6c8-fe61cb097aae-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:47.207523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.207502    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-maas-api-tls" pod "5f211138-01d6-408c-a6c8-fe61cb097aae" (UID: "5f211138-01d6-408c-a6c8-fe61cb097aae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:47.207611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.207568    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f211138-01d6-408c-a6c8-fe61cb097aae-kube-api-access-qbqdz" pod "5f211138-01d6-408c-a6c8-fe61cb097aae" (UID: "5f211138-01d6-408c-a6c8-fe61cb097aae"). InnerVolumeSpecName "kube-api-access-qbqdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:47.207674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.207655    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-metrics-certs" pod "5f211138-01d6-408c-a6c8-fe61cb097aae" (UID: "5f211138-01d6-408c-a6c8-fe61cb097aae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:10:47.307962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.307879    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:47.307962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.307912    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbqdz\" (UniqueName: \"kubernetes.io/projected/5f211138-01d6-408c-a6c8-fe61cb097aae-kube-api-access-qbqdz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:47.307962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.307923    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f211138-01d6-408c-a6c8-fe61cb097aae-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:47.468286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.468240    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" path="/var/lib/kubelet/pods/b76cd7cc-1bc1-401d-8a81-c1dd02e6c338/volumes"
Nov 03 20:10:47.468803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:47.468784    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8bddb75-dbbc-4907-8c40-0ca54c150a92" path="/var/lib/kubelet/pods/c8bddb75-dbbc-4907-8c40-0ca54c150a92/volumes"
Nov 03 20:10:48.022911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:48.022872    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k" event={"ID":"5f211138-01d6-408c-a6c8-fe61cb097aae","Type":"ContainerDied","Data":"f1533effc440362d2c2ede1cdfdf67cef2994ed3dde6d6726cafb673e5a4e244"}
Nov 03 20:10:48.023113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:48.022943    5282 scope.go:122] "RemoveContainer" containerID="9e8c8975b96452486fb50b5df621800f40c27b6e94589ffd3014735062c9301d"
Nov 03 20:10:48.023113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:48.022892    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"
Nov 03 20:10:48.045717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:48.045693    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"]
Nov 03 20:10:48.050072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:48.050050    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d441dd-7f8455c5cd-l9r6k"]
Nov 03 20:10:48.102447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:48.102418    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:48.429512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:48.429418    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:49.459459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:49.459417    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f211138-01d6-408c-a6c8-fe61cb097aae" path="/var/lib/kubelet/pods/5f211138-01d6-408c-a6c8-fe61cb097aae/volumes"
Nov 03 20:10:49.615658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:49.615629    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:51.042925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:51.042889    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:52.062043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:52.062003    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:52.571986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:52.571951    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:52.917274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:52.907589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:53.055157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:53.055120    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:53.212317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:53.208732    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"]
Nov 03 20:10:53.212317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:53.208992    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" containerID="cri-o://ef9d0a2c57221dde360adfa62464de720d2da2f58f4c69b00ace66277c92aabd" gracePeriod=30
Nov 03 20:10:53.226619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:53.226580    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:54.091377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:54.091342    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:54.621663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:54.621626    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:54.983265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:54.983231    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:55.154221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:55.147687    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:55.342229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:55.342121    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:55.963999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:55.963964    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:56.714771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:56.714733    5282 patch_prober.go:29] interesting pod/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:10:56.723396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:56.723352    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:10:56.873629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:56.873601    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:57.385581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:57.385549    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"]
Nov 03 20:10:57.517850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:57.517816    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:57.678325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:57.677815    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:58.366908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.366869    5282 generic.go:356] "Generic (PLEG): container finished" podID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerID="ef9d0a2c57221dde360adfa62464de720d2da2f58f4c69b00ace66277c92aabd" exitCode=0
Nov 03 20:10:58.367084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.366927    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" event={"ID":"2e13f182-06e1-4122-922b-ebc4ebeca7ce","Type":"ContainerDied","Data":"ef9d0a2c57221dde360adfa62464de720d2da2f58f4c69b00ace66277c92aabd"}
Nov 03 20:10:58.367084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.367027    5282 scope.go:122] "RemoveContainer" containerID="773e386b3e898f7d23d056b40466205291ad5913f6ade00184935326b1639941"
Nov 03 20:10:58.389635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.389475    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:58.672381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.672346    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:10:58.728139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728064    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-credential-socket\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.728408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728173    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-podinfo\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.728408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728224    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-envoy\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.728408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728254    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istiod-ca-cert\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.728408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728277    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:58.728816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728749    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istiod-ca-cert" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:10:58.728914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728758    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-socket\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.728973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728942    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-socket" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:58.728973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.728945    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-kube-api-access-l42k2\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-kube-api-access-l42k2\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.729088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.729017    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-token\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-token\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.730289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.729167    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-data\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.730289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.729189    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-credential-socket" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:58.730289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.729364    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-certs\") pod \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\" (UID: \"2e13f182-06e1-4122-922b-ebc4ebeca7ce\") "
Nov 03 20:10:58.730874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.730847    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-certs" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:58.731008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.730959    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.731008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.730983    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.731008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.730996    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.731176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.731046    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-data" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:58.731687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.731660    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-kube-api-access-l42k2" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "kube-api-access-l42k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:58.732730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.732697    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-podinfo" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:10:58.732811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.732755    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-token" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:10:58.734676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.734651    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-envoy" pod "2e13f182-06e1-4122-922b-ebc4ebeca7ce" (UID: "2e13f182-06e1-4122-922b-ebc4ebeca7ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:10:58.832704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.832671    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l42k2\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-kube-api-access-l42k2\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.832704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.832701    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.832704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.832711    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.832942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.832720    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.832942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.832728    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.832942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.832737    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e13f182-06e1-4122-922b-ebc4ebeca7ce-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:10:58.853676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:58.853641    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:59.034236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.028773    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:59.370263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.370161    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:59.383080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.383039    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8" event={"ID":"2e13f182-06e1-4122-922b-ebc4ebeca7ce","Type":"ContainerDied","Data":"16719a2696d87382ef04375e76681e593674e6706481cc257f5112e313432025"}
Nov 03 20:10:59.383428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.383375    5282 scope.go:122] "RemoveContainer" containerID="ef9d0a2c57221dde360adfa62464de720d2da2f58f4c69b00ace66277c92aabd"
Nov 03 20:10:59.383729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.383682    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"
Nov 03 20:10:59.423960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.423924    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"]
Nov 03 20:10:59.428290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.428256    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d441dd-openshift-default-64bfb6d74c-6j4d8"]
Nov 03 20:10:59.457983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.457951    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" path="/var/lib/kubelet/pods/2e13f182-06e1-4122-922b-ebc4ebeca7ce/volumes"
Nov 03 20:10:59.697411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.697317    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:10:59.845975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:10:59.845945    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:00.134580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:00.134548    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:00.574221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:00.544340    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:00.870068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:00.869969    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:01.306210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:01.306157    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:01.618564 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:01.618487    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:11:01.981250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:01.981179    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:02.386322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.386240    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2" podUID="f30bd10e-d74d-4f45-b0b5-21ca277c459f" containerName="payload-pre-processing" containerID="cri-o://4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f" gracePeriod=25
Nov 03 20:11:02.580471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.580440    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:02.817249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.810544    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:11:02.904097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.904063    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30bd10e-d74d-4f45-b0b5-21ca277c459f-kube-api-access-wj7c5\" (UniqueName: \"kubernetes.io/projected/f30bd10e-d74d-4f45-b0b5-21ca277c459f-kube-api-access-wj7c5\") pod \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") "
Nov 03 20:11:02.904318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.904122    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30bd10e-d74d-4f45-b0b5-21ca277c459f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30bd10e-d74d-4f45-b0b5-21ca277c459f-plugins-config-volume\") pod \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\" (UID: \"f30bd10e-d74d-4f45-b0b5-21ca277c459f\") "
Nov 03 20:11:02.904623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.904591    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30bd10e-d74d-4f45-b0b5-21ca277c459f-plugins-config-volume" pod "f30bd10e-d74d-4f45-b0b5-21ca277c459f" (UID: "f30bd10e-d74d-4f45-b0b5-21ca277c459f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:02.905029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.905003    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30bd10e-d74d-4f45-b0b5-21ca277c459f-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:02.906441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.906402    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30bd10e-d74d-4f45-b0b5-21ca277c459f-kube-api-access-wj7c5" pod "f30bd10e-d74d-4f45-b0b5-21ca277c459f" (UID: "f30bd10e-d74d-4f45-b0b5-21ca277c459f"). InnerVolumeSpecName "kube-api-access-wj7c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:02.925816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.925790    5282 generic.go:356] "Generic (PLEG): container finished" podID="f30bd10e-d74d-4f45-b0b5-21ca277c459f" containerID="4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f" exitCode=0
Nov 03 20:11:02.925932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.925837    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"
Nov 03 20:11:02.925932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.925875    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2" event={"ID":"f30bd10e-d74d-4f45-b0b5-21ca277c459f","Type":"ContainerDied","Data":"4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f"}
Nov 03 20:11:02.926055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.925936    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2" event={"ID":"f30bd10e-d74d-4f45-b0b5-21ca277c459f","Type":"ContainerDied","Data":"ce6080655113d10426e4cd969a8f15a87e4f11dd55567f1f9235bedb01ae19c5"}
Nov 03 20:11:02.926055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.925966    5282 scope.go:122] "RemoveContainer" containerID="4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f"
Nov 03 20:11:02.936822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.936805    5282 scope.go:122] "RemoveContainer" containerID="4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f"
Nov 03 20:11:02.937096 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:02.937072    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f\": container with ID starting with 4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f not found: ID does not exist" containerID="4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f"
Nov 03 20:11:02.937146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.937110    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f"} err="failed to get container status \"4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f\": rpc error: code = NotFound desc = could not find container \"4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f\": container with ID starting with 4b6eb8cbd868f049bfe92b61220ef2fdf46627e739d703803a8b383519ebff3f not found: ID does not exist"
Nov 03 20:11:02.950620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.950601    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"]
Nov 03 20:11:02.957361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:02.957339    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1962f9-55df7dc9c9-4pcn2"]
Nov 03 20:11:03.006892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.006860    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj7c5\" (UniqueName: \"kubernetes.io/projected/f30bd10e-d74d-4f45-b0b5-21ca277c459f-kube-api-access-wj7c5\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:03.214969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.214923    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:03.408890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.408786    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:03.458524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.458485    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30bd10e-d74d-4f45-b0b5-21ca277c459f" path="/var/lib/kubelet/pods/f30bd10e-d74d-4f45-b0b5-21ca277c459f/volumes"
Nov 03 20:11:03.765278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.765244    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:03.778297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.778258    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"]
Nov 03 20:11:03.779822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.779796    5282 cpu_manager.go:414] "containerMap: removing container" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy"
Nov 03 20:11:03.779987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.779972    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy"
Nov 03 20:11:03.780129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.780116    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f30bd10e-d74d-4f45-b0b5-21ca277c459f" containerName="payload-pre-processing"
Nov 03 20:11:03.780629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.780596    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30bd10e-d74d-4f45-b0b5-21ca277c459f" containerName="payload-pre-processing"
Nov 03 20:11:03.780792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.780778    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c8bddb75-dbbc-4907-8c40-0ca54c150a92" containerName="payload-processing"
Nov 03 20:11:03.781056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.781012    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bddb75-dbbc-4907-8c40-0ca54c150a92" containerName="payload-processing"
Nov 03 20:11:03.797421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797396    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" containerName="payload-pre-processing"
Nov 03 20:11:03.797600 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797574    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" containerName="payload-pre-processing"
Nov 03 20:11:03.797781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797644    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5f211138-01d6-408c-a6c8-fe61cb097aae" containerName="maas-api"
Nov 03 20:11:03.797781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797653    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f211138-01d6-408c-a6c8-fe61cb097aae" containerName="maas-api"
Nov 03 20:11:03.797937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797915    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f211138-01d6-408c-a6c8-fe61cb097aae" containerName="maas-api"
Nov 03 20:11:03.797937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797934    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy"
Nov 03 20:11:03.798071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797950    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy"
Nov 03 20:11:03.798071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797968    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30bd10e-d74d-4f45-b0b5-21ca277c459f" containerName="payload-pre-processing"
Nov 03 20:11:03.798071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797982    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bddb75-dbbc-4907-8c40-0ca54c150a92" containerName="payload-processing"
Nov 03 20:11:03.798071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.797995    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76cd7cc-1bc1-401d-8a81-c1dd02e6c338" containerName="payload-pre-processing"
Nov 03 20:11:03.805256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.805233    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"]
Nov 03 20:11:03.805436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.805265    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.812339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.812316    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49fb2b-openshift-default-dockercfg-w4fn6\""
Nov 03 20:11:03.833824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.833798    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-envoy\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.833967    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-kube-api-access-rmgm9\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.834047    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-token\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.834191    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-credential-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.834361    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-podinfo\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.834470    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.834529    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istiod-ca-cert\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.834576    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-certs\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.834741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.834682    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-data\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.902180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.902142    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:03.936695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.936646    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-envoy\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.936909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.936765    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-kube-api-access-rmgm9\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.936909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.936826    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-token\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.936909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.936879    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-credential-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.937120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.936923    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-podinfo\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.937120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.936973    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.937120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.936999    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istiod-ca-cert\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.937120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.937053    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-certs\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.937120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.937099    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-data\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:03.978689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.978640    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"]
Nov 03 20:11:04.002237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:03.997466    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-credential-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.193133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.192816    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:04.230219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.198938    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-socket\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.230219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.199493    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-certs\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.230219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.200059    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istiod-ca-cert\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.230219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.200288    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-data\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.346219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.324345    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-podinfo\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.346219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.332954    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:04.372293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.355535    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-envoy\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.477266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.453120    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-token\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.567219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.486138    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:04.567219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.487571    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-kube-api-access-rmgm9\") pod \"e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") " pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.615312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.584608    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:04.619698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:04.619644    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:05.549384 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:11:05.549347    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab761e2b_c2a0_4a41_a8ec_efdc9df389e2.slice/crio-effd585421a4dd6ba6d04eaf171f0321f06acc9562aef961790cc633473f392a WatchSource:0}: Error finding container effd585421a4dd6ba6d04eaf171f0321f06acc9562aef961790cc633473f392a: Status 404 returned error can't find the container with id effd585421a4dd6ba6d04eaf171f0321f06acc9562aef961790cc633473f392a
Nov 03 20:11:05.550119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:05.550087    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"]
Nov 03 20:11:05.552453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:05.552417    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:05.552519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:05.552485    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:05.552566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:05.552515    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:05.708801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:05.708746    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:06.318558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:06.318519    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" event={"ID":"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2","Type":"ContainerStarted","Data":"bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6"}
Nov 03 20:11:06.318766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:06.318577    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" event={"ID":"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2","Type":"ContainerStarted","Data":"effd585421a4dd6ba6d04eaf171f0321f06acc9562aef961790cc633473f392a"}
Nov 03 20:11:06.340310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:06.340034    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podStartSLOduration=3.340020441 podStartE2EDuration="3.340020441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:06.339486819 +0000 UTC m=+1859.008688287" watchObservedRunningTime="2026-11-03 20:11:06.340020441 +0000 UTC m=+1859.009221921"
Nov 03 20:11:06.399990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:06.399743    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:06.596290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:06.588777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:06.608221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:06.606807    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:06.998222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:06.995430    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:07.152231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:07.149790    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:07.338026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:07.337942    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:07.339516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:07.339492    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:11:07.951477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:07.951439    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:08.499734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:08.499701    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:08.662219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:08.662141    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:08.799269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:08.798719    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:08.954262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:08.943985    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:08.986322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:08.986273    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w" podUID="0bccfe06-8c2d-44e2-aa95-921392f4b108" containerName="payload-pre-processing" containerID="cri-o://7bf066e4780e2bb76172a1232f40f7d39f026a62c62666c22164b08c167faa87" gracePeriod=25
Nov 03 20:11:09.360671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.359670    5282 generic.go:356] "Generic (PLEG): container finished" podID="0bccfe06-8c2d-44e2-aa95-921392f4b108" containerID="7bf066e4780e2bb76172a1232f40f7d39f026a62c62666c22164b08c167faa87" exitCode=0
Nov 03 20:11:09.361789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.361537    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w" event={"ID":"0bccfe06-8c2d-44e2-aa95-921392f4b108","Type":"ContainerDied","Data":"7bf066e4780e2bb76172a1232f40f7d39f026a62c62666c22164b08c167faa87"}
Nov 03 20:11:09.361789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.361632    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w" event={"ID":"0bccfe06-8c2d-44e2-aa95-921392f4b108","Type":"ContainerDied","Data":"d03f3abd5ac114773687f2014dc7e6e1b7c4eaa53e7c967b1ef54f9473ed8d88"}
Nov 03 20:11:09.361789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.361665    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d03f3abd5ac114773687f2014dc7e6e1b7c4eaa53e7c967b1ef54f9473ed8d88"
Nov 03 20:11:09.411315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.411288    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:11:09.513513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.513476    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bccfe06-8c2d-44e2-aa95-921392f4b108-kube-api-access-klcrm\" (UniqueName: \"kubernetes.io/projected/0bccfe06-8c2d-44e2-aa95-921392f4b108-kube-api-access-klcrm\") pod \"0bccfe06-8c2d-44e2-aa95-921392f4b108\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") "
Nov 03 20:11:09.513666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.513532    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bccfe06-8c2d-44e2-aa95-921392f4b108-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bccfe06-8c2d-44e2-aa95-921392f4b108-plugins-config-volume\") pod \"0bccfe06-8c2d-44e2-aa95-921392f4b108\" (UID: \"0bccfe06-8c2d-44e2-aa95-921392f4b108\") "
Nov 03 20:11:09.514046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.514016    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bccfe06-8c2d-44e2-aa95-921392f4b108-plugins-config-volume" pod "0bccfe06-8c2d-44e2-aa95-921392f4b108" (UID: "0bccfe06-8c2d-44e2-aa95-921392f4b108"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:09.514908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.514881    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bccfe06-8c2d-44e2-aa95-921392f4b108-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:09.516114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.516085    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bccfe06-8c2d-44e2-aa95-921392f4b108-kube-api-access-klcrm" pod "0bccfe06-8c2d-44e2-aa95-921392f4b108" (UID: "0bccfe06-8c2d-44e2-aa95-921392f4b108"). InnerVolumeSpecName "kube-api-access-klcrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:09.616838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.616754    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klcrm\" (UniqueName: \"kubernetes.io/projected/0bccfe06-8c2d-44e2-aa95-921392f4b108-kube-api-access-klcrm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:09.739036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.739001    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:09.863379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:09.863347    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:10.077220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:10.051099    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:10.504221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:10.482100    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"
Nov 03 20:11:10.572248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:10.572066    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"]
Nov 03 20:11:10.603370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:10.603305    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b9dc48-56b9bcd659-k5d4w"]
Nov 03 20:11:10.899726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:10.899631    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:11.549775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:11.549733    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bccfe06-8c2d-44e2-aa95-921392f4b108" path="/var/lib/kubelet/pods/0bccfe06-8c2d-44e2-aa95-921392f4b108/volumes"
Nov 03 20:11:11.777983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:11.777949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:11.913175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:11.913067    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:12.138553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.138512    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:12.139011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.138992    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"]
Nov 03 20:11:12.140131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.140111    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0bccfe06-8c2d-44e2-aa95-921392f4b108" containerName="payload-pre-processing"
Nov 03 20:11:12.140219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.140135    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bccfe06-8c2d-44e2-aa95-921392f4b108" containerName="payload-pre-processing"
Nov 03 20:11:12.140219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.140174    5282 cpu_manager.go:414] "containerMap: removing container" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy"
Nov 03 20:11:12.140219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.140182    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e13f182-06e1-4122-922b-ebc4ebeca7ce" containerName="istio-proxy"
Nov 03 20:11:12.140440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.140427    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bccfe06-8c2d-44e2-aa95-921392f4b108" containerName="payload-pre-processing"
Nov 03 20:11:12.144477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.144457    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:12.149586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.149547    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-acd98d21\""
Nov 03 20:11:12.154758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.154731    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"]
Nov 03 20:11:12.278479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.278395    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:12.298224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.297094    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2wt\" (UniqueName: \"kubernetes.io/projected/7252bd12-b6ea-4a9b-b60b-f53a705a641d-kube-api-access-rl2wt\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:12.298224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.297231    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7252bd12-b6ea-4a9b-b60b-f53a705a641d-plugins-config-volume\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:12.410218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.409234    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2wt\" (UniqueName: \"kubernetes.io/projected/7252bd12-b6ea-4a9b-b60b-f53a705a641d-kube-api-access-rl2wt\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:12.410218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.409349    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7252bd12-b6ea-4a9b-b60b-f53a705a641d-plugins-config-volume\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:12.410466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.410312    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7252bd12-b6ea-4a9b-b60b-f53a705a641d-plugins-config-volume\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:12.487225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.485862    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2wt\" (UniqueName: \"kubernetes.io/projected/7252bd12-b6ea-4a9b-b60b-f53a705a641d-kube-api-access-rl2wt\") pod \"payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") " pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:12.767233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:12.765567    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:13.188413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:13.187065    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:13.215224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:13.213276    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"]
Nov 03 20:11:13.656638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:13.655174    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc" event={"ID":"7252bd12-b6ea-4a9b-b60b-f53a705a641d","Type":"ContainerStarted","Data":"14daa48c7d1d6af450d6433427be1c44d4272368d233f1d8bfa1b39e5ff03948"}
Nov 03 20:11:13.656638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:13.655265    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc" event={"ID":"7252bd12-b6ea-4a9b-b60b-f53a705a641d","Type":"ContainerStarted","Data":"7784333f7291ab91469e94f435f38646d37b217bd8b97b31b03d82aa4cc81092"}
Nov 03 20:11:13.656638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:13.656533    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:13.694694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:13.693893    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc" podStartSLOduration=1.693870241 podStartE2EDuration="1.693870241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:13.680991962 +0000 UTC m=+1866.350193433" watchObservedRunningTime="2026-11-03 20:11:13.693870241 +0000 UTC m=+1866.363071710"
Nov 03 20:11:13.875798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:13.875762    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:11:14.280747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:14.247821    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:14.406138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:14.392013    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:14.659370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:14.659285    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:15.265410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:15.265376    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:17.502225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:17.501918    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:17.960293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:17.959381    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"]
Nov 03 20:11:18.110485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.110444    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"]
Nov 03 20:11:18.110841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.110819    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190472    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190558    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7xm\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-kube-api-access-8h7xm\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190655    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190701    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190761    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190850    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190932    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.190974    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.210386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.191107    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.293551    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.293625    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.293690    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.293780    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.293870    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.293914    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.294046    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.294166    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.294269    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7xm\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-kube-api-access-8h7xm\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.303140    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-certs\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.303442    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.303693    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-credential-socket\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.304362    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istiod-ca-cert\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.309935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.309864    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-data\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.341311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.319578    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-podinfo\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.341903 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.341819    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-envoy\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.355309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.353020    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-token\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.419222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.409903    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7xm\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-kube-api-access-8h7xm\") pod \"e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") " pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.565324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.565269    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"]
Nov 03 20:11:18.580429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.580397    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"]
Nov 03 20:11:18.580767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.580702    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:18.716219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.706247    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:18.716219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.708284    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:18.952220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.921774    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.032889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.032852    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.033209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.033167    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.033481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.033458    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj9xz\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-kube-api-access-bj9xz\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.033688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.033669    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.033818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.033801    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.034083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.034061    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.035766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:18.965254    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:19.035766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.013471    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:19.036036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.036014    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.036271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.036246    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.087242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.085723    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:19.114475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.114369    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.124980    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.125128    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.139566    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.139725    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.139766    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.139841    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.139953    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9xz\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-kube-api-access-bj9xz\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.140039    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.140079    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.140251    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.140356    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.157226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.140956    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.181222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.168444    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.181222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.168519    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.181222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.169287    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.182293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.182267    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.229844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.087674    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:19.245223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.242183    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.274419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.260316    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:19.276067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.276033    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9xz\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-kube-api-access-bj9xz\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.324133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.322912    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.475521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.468805    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.503831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.503551    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:19.722513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.704392    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:19.776430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.776365    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:11:19.789321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:19.789274    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:11:20.035545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.035497    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:20.039476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.039440    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:11:20.232562 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.232519    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:20.233163 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.232986    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 03 20:11:20.239227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.239168    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:11:20.276467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.276425    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:20.307352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.307263    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:11:20.333217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.331071    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:11:20.333217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.331467    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:11:20.333217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.331532    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:11:20.333217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.331571    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:11:20.352090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.352052    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:20.451225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.443256    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:11:20.458048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.458020    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:11:20.498002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.494410    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:11:20.536410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.536375    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:11:20.571028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.569695    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:11:20.858623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.856976    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:20.858623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.857657    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:20.858623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.857891    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:20.943437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.938953    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"]
Nov 03 20:11:20.943437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.939134    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"]
Nov 03 20:11:20.944572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.944418    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:20.944697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.944581    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:20.944697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:20.944685    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:21.045702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.044042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:21.268284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.268238    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:21.410380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.410312    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:21.507222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.504862    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" event={"ID":"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9","Type":"ContainerStarted","Data":"90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1"}
Nov 03 20:11:21.507222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.505326    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" event={"ID":"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9","Type":"ContainerStarted","Data":"1eb81fe044b32304ab54f1b35200424c76660143682a9c0bf08096d8eba0fc4b"}
Nov 03 20:11:21.518223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.517289    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" event={"ID":"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c","Type":"ContainerStarted","Data":"c59e758ff4635fe43931403b6cf45960cca9b1c7b87476ed327666c3868e9bb8"}
Nov 03 20:11:21.518223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.517370    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" event={"ID":"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c","Type":"ContainerStarted","Data":"9d92d32b5a8c385badc258caa811cd511455d818bba50a4d9111423b4ddef8f5"}
Nov 03 20:11:21.540420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.540033    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podStartSLOduration=4.5400147969999995 podStartE2EDuration="4.540014797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:21.535131307 +0000 UTC m=+1874.204332779" watchObservedRunningTime="2026-11-03 20:11:21.540014797 +0000 UTC m=+1874.209216272"
Nov 03 20:11:21.600224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.580107    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podStartSLOduration=3.580087891 podStartE2EDuration="3.580087891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:21.578028981 +0000 UTC m=+1874.247230452" watchObservedRunningTime="2026-11-03 20:11:21.580087891 +0000 UTC m=+1874.249289363"
Nov 03 20:11:21.763451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.763362    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:21.884399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.884355    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:11:21.884897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.884424    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:21.929521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:21.929487    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:22.514279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:22.510746    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:22.514279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:22.510870    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:22.514279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:22.510916    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:22.765935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:22.765849    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:22.766091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:22.765970    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:22.837272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:22.837229    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:23.002317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:22.998159    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:23.356233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:23.347857    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:23.527145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:23.527095    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:23.527432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:23.527393    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:23.779387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:23.766606    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:23.779387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:23.766673    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:23.880912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:23.880874    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"]
Nov 03 20:11:23.881446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:23.881409    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" containerID="cri-o://06f13aae047020c2fe68f8edbdda4e09e8e4378fdfbc36544cdd440c417ecfc1" gracePeriod=30
Nov 03 20:11:24.138070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.137986    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"]
Nov 03 20:11:24.140503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.140467    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy" containerID="cri-o://5dbb3e1dd37cd663954cff220639fb00d53a313609c972f24b9c442fbb053117" gracePeriod=30
Nov 03 20:11:24.472252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.472125    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:24.519448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.519390    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:24.519649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.519478    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:24.524349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.524313    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:24.591220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.583774    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:24.671211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.647583    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:24.739078 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.718644    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:24.739078 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.733294    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:24.753404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.753357    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:11:24.781392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.781349    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:24.895711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.895643    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:11:24.962547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:24.962071    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:25.088046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.078752    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:25.088046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.078819    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:25.269518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.269484    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:25.527855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.518545    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:25.582220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.559424    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:25.601219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.584498    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:25.638416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.638374    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:25.661643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.661592    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:25.681834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.681793    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:25.682156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.682130    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:11:25.731353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.731322    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:25.791324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.769130    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:25.791324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.769211    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:25.986456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.973815    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:11:25.986456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:25.982366    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:26.142338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.136371    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:11:26.148724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.148680    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:11:26.203761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.203731    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:11:26.213386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.213352    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:11:26.221753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.213665    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:11:26.224210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.224161    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:11:26.225031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.182935    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:26.374221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.370066    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:11:26.557951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.426517    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:26.565856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.558425    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:26.565856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.565370    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:26.668301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.575486    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:26.668301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.581716    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:26.668301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.581762    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:26.668301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.583791    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:11:26.668301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.587173    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:11:26.745596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.745546    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:26.754975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.754934    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:11:26.755140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.755001    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:11:26.762775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.762740    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:26.771478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.771445    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:26.771609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.771499    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:26.861310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:26.856373    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:11:27.518656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:27.518616    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:27.519285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:27.519249    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:27.765535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:27.765242    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:27.765535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:27.765284    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:27.936809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:27.936730    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:28.508073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:28.507610    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:28.508073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:28.507675    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:28.736959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:28.736920    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:28.779282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:28.777344    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:28.779282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:28.777405    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:29.530561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:29.529592    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 20:11:29.530561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:29.529654    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 20:11:29.662223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:29.650866    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:29.846140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:29.838734    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:29.846140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:29.838798    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:29.957992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:29.957943    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:30.762506 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:30.736781    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:30.946034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:30.945992    5282 generic.go:356] "Generic (PLEG): container finished" podID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerID="da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b" exitCode=0
Nov 03 20:11:30.947010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:30.946979    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" event={"ID":"27507f86-3733-48d7-9223-c5fb07e94f9e","Type":"ContainerDied","Data":"da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b"}
Nov 03 20:11:30.949838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:30.949784    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:30.981482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:30.981451    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:31.030699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.030592    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 03 20:11:31.081658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.081614    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 03 20:11:31.082391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.080728    5282 scope.go:122] "RemoveContainer" containerID="da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b"
Nov 03 20:11:31.117687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.117657    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:31.253874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.253833    5282 generic.go:356] "Generic (PLEG): container finished" podID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerID="5dbb3e1dd37cd663954cff220639fb00d53a313609c972f24b9c442fbb053117" exitCode=0
Nov 03 20:11:31.254356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.254316    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" event={"ID":"197a300d-62a7-48b6-9b14-0c479de0f8fd","Type":"ContainerDied","Data":"5dbb3e1dd37cd663954cff220639fb00d53a313609c972f24b9c442fbb053117"}
Nov 03 20:11:31.272462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.272416    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:31.275383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.275333    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:31.275586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.275556    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:31.275702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.275679    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:31.281603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.280858    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:11:31.283446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.283424    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:11:31.289376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.289265    5282 generic.go:356] "Generic (PLEG): container finished" podID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerID="06f13aae047020c2fe68f8edbdda4e09e8e4378fdfbc36544cdd440c417ecfc1" exitCode=0
Nov 03 20:11:31.289376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.289330    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" event={"ID":"4ed66650-0337-4a97-9540-4c584d1ddab9","Type":"ContainerDied","Data":"06f13aae047020c2fe68f8edbdda4e09e8e4378fdfbc36544cdd440c417ecfc1"}
Nov 03 20:11:31.289576 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.289426    5282 scope.go:122] "RemoveContainer" containerID="06f13aae047020c2fe68f8edbdda4e09e8e4378fdfbc36544cdd440c417ecfc1"
Nov 03 20:11:31.345537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.341713    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:31.345852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.345561    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:11:31.459373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.448612    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-credential-socket\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.459373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.448724    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-token\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-token\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.476067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.476034    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-credential-socket" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.560569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.560461    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-token" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.565464    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-certs\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.565599    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-token\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-token\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.565667    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-podinfo\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.565729    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-podinfo\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.565762    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197a300d-62a7-48b6-9b14-0c479de0f8fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197a300d-62a7-48b6-9b14-0c479de0f8fd-istiod-ca-cert\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.565933    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-data\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.566003    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-kube-api-access-wwvbf\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-kube-api-access-wwvbf\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.566075    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-certs\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.566119    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-socket\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.566230    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-data\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.566346    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-envoy\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.567666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.566417    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed66650-0337-4a97-9540-4c584d1ddab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed66650-0337-4a97-9540-4c584d1ddab9-istiod-ca-cert\") pod \"4ed66650-0337-4a97-9540-4c584d1ddab9\" (UID: \"4ed66650-0337-4a97-9540-4c584d1ddab9\") "
Nov 03 20:11:31.568856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.568491    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-data" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.584833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.584788    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197a300d-62a7-48b6-9b14-0c479de0f8fd-istiod-ca-cert" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:31.594795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.594760    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-socket" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.599322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.599285    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-certs" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.599985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.599948    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-data" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.602033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.602007    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-certs" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.610602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.610576    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed66650-0337-4a97-9540-4c584d1ddab9-istiod-ca-cert" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:31.637961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.637455    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.637961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.637489    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.646976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.645707    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-podinfo" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:31.646976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.646354    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-kube-api-access-wwvbf" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "kube-api-access-wwvbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:31.648880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.648831    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-envoy" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.649620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.649598    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-podinfo" pod "4ed66650-0337-4a97-9540-4c584d1ddab9" (UID: "4ed66650-0337-4a97-9540-4c584d1ddab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:31.652239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.652184    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-token" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:31.717588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.716170    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"]
Nov 03 20:11:31.718914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.718391    5282 cpu_manager.go:414] "containerMap: removing container" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy"
Nov 03 20:11:31.718914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.718416    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy"
Nov 03 20:11:31.718914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.718496    5282 cpu_manager.go:414] "containerMap: removing container" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy"
Nov 03 20:11:31.718914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.718506    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy"
Nov 03 20:11:31.718914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.718737    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" containerName="istio-proxy"
Nov 03 20:11:31.718914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.718764    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy"
Nov 03 20:11:31.727577 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:31.724974    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:11:31.727577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.727158    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"]
Nov 03 20:11:31.727577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.727363    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.734400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.733684    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-49fb2b-serving-cert\""
Nov 03 20:11:31.747162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.747121    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-envoy" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.749447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.749425    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-envoy\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.749563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.749495    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-kube-api-access-qc86c\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-kube-api-access-qc86c\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.749563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.749548    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-credential-socket\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.749675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.749592    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-socket\") pod \"197a300d-62a7-48b6-9b14-0c479de0f8fd\" (UID: \"197a300d-62a7-48b6-9b14-0c479de0f8fd\") "
Nov 03 20:11:31.750164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.749975    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-socket" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.750715 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750695    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750719    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750733    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197a300d-62a7-48b6-9b14-0c479de0f8fd-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750746    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750760    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvbf\" (UniqueName: \"kubernetes.io/projected/4ed66650-0337-4a97-9540-4c584d1ddab9-kube-api-access-wwvbf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750774    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750788    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750801    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.750819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750814    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed66650-0337-4a97-9540-4c584d1ddab9-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.751340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750830    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed66650-0337-4a97-9540-4c584d1ddab9-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.751340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750843    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.751340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750857    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.751340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750872    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.751340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.750885    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.751340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.751249    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-credential-socket" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:31.760158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.760118    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-kube-api-access-qc86c" pod "197a300d-62a7-48b6-9b14-0c479de0f8fd" (UID: "197a300d-62a7-48b6-9b14-0c479de0f8fd"). InnerVolumeSpecName "kube-api-access-qc86c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:31.772326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.772303    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:31.773415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.773356    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:31.774493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.774476    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:31.853810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.853600    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d58810-f6b1-43cf-893f-37bec467763b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.854798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.854760    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzd9\" (UniqueName: \"kubernetes.io/projected/95d58810-f6b1-43cf-893f-37bec467763b-kube-api-access-qnzd9\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.854980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.854843    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-metrics-certs\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.854980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.854895    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-maas-api-tls\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.855108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.855093    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc86c\" (UniqueName: \"kubernetes.io/projected/197a300d-62a7-48b6-9b14-0c479de0f8fd-kube-api-access-qc86c\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.855167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.855110    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197a300d-62a7-48b6-9b14-0c479de0f8fd-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:31.937522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.937488    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:11:31.937717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.937542    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:31.956663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.956635    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzd9\" (UniqueName: \"kubernetes.io/projected/95d58810-f6b1-43cf-893f-37bec467763b-kube-api-access-qnzd9\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.957103 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.956707    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-metrics-certs\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.957103 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.956757    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-maas-api-tls\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.957103 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.957070    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d58810-f6b1-43cf-893f-37bec467763b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.958259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.958235    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d58810-f6b1-43cf-893f-37bec467763b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.959573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.959550    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-metrics-certs\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.959662 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.959586    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-maas-api-tls\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.964113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.964093    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzd9\" (UniqueName: \"kubernetes.io/projected/95d58810-f6b1-43cf-893f-37bec467763b-kube-api-access-qnzd9\") pod \"maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:31.989658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:31.989615    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:32.054297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.053990    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:32.305849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.305676    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"
Nov 03 20:11:32.305849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.305699    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks" event={"ID":"197a300d-62a7-48b6-9b14-0c479de0f8fd","Type":"ContainerDied","Data":"67a57abdebfd36af4ad9fce9d6001a41a37aedee3f3982530393ad35a5ea0201"}
Nov 03 20:11:32.305849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.305766    5282 scope.go:122] "RemoveContainer" containerID="5dbb3e1dd37cd663954cff220639fb00d53a313609c972f24b9c442fbb053117"
Nov 03 20:11:32.310295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.309655    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz" event={"ID":"4ed66650-0337-4a97-9540-4c584d1ddab9","Type":"ContainerDied","Data":"7fda505ea4d91412f40734491ce1149beabbc5fc5c953b6255e6fc1d28e771e7"}
Nov 03 20:11:32.310295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.309755    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"
Nov 03 20:11:32.317340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.316257    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" event={"ID":"27507f86-3733-48d7-9223-c5fb07e94f9e","Type":"ContainerStarted","Data":"2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e"}
Nov 03 20:11:32.359479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.359424    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"]
Nov 03 20:11:32.364917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.364853    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-tt6ks"]
Nov 03 20:11:32.374358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.374305    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"]
Nov 03 20:11:32.441372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.441304    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-p8hjz"]
Nov 03 20:11:32.526667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.526629    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"]
Nov 03 20:11:32.609484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.609315    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:32.751944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.751901    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:32.878150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:32.875598    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:33.115648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.115551    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:11:33.148225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.118288    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 20:11:33.148225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.127396    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 20:11:33.268247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.255259    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:33.768750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.768706    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197a300d-62a7-48b6-9b14-0c479de0f8fd" path="/var/lib/kubelet/pods/197a300d-62a7-48b6-9b14-0c479de0f8fd/volumes"
Nov 03 20:11:33.769818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.769787    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed66650-0337-4a97-9540-4c584d1ddab9" path="/var/lib/kubelet/pods/4ed66650-0337-4a97-9540-4c584d1ddab9/volumes"
Nov 03 20:11:33.770432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.770403    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z" event={"ID":"95d58810-f6b1-43cf-893f-37bec467763b","Type":"ContainerStarted","Data":"6bfbf36f3074b0c13c44b1e97cf0bd067525776c70592f31f52a8ab02df399c2"}
Nov 03 20:11:33.770545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.770476    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"]
Nov 03 20:11:33.822392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.822356    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"]
Nov 03 20:11:33.822533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.822484    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.826698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.826668    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:11:33.826862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.826840    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:11:33.832758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.832736    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.832923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.832909    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.833252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.833236    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9gz\" (UniqueName: \"kubernetes.io/projected/9d356215-1ebc-4bd1-85ed-16245f324280-kube-api-access-kp9gz\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.935086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.935042    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.935322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.935108    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.935397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.935324    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9gz\" (UniqueName: \"kubernetes.io/projected/9d356215-1ebc-4bd1-85ed-16245f324280-kube-api-access-kp9gz\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.935854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.935827    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.937491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.937467    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:33.959807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:33.959782    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9gz\" (UniqueName: \"kubernetes.io/projected/9d356215-1ebc-4bd1-85ed-16245f324280-kube-api-access-kp9gz\") pod \"maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:34.089642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.089552    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:34.145258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.118511    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 20:11:34.147746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.147608    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 20:11:34.148602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.148492    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:34.311382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.310972    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:11:34.311382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.311032    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:34.545398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.545362    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"]
Nov 03 20:11:34.772092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.771999    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z" event={"ID":"95d58810-f6b1-43cf-893f-37bec467763b","Type":"ContainerStarted","Data":"4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c"}
Nov 03 20:11:34.772311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.772106    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:34.797868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.797784    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf" event={"ID":"9d356215-1ebc-4bd1-85ed-16245f324280","Type":"ContainerStarted","Data":"bf3e93cf0e8fa56fccbc30673733850db0b77bc25cbd8a86672eea7ee8882c1f"}
Nov 03 20:11:34.811394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:34.811340    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z" podStartSLOduration=4.870635168 podStartE2EDuration="5.811323198s" totalImagesPullingTime="940.68803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:34.810566342 +0000 UTC m=+1887.479767815" watchObservedRunningTime="2026-11-03 20:11:34.811323198 +0000 UTC m=+1887.480524668"
Nov 03 20:11:35.060280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:35.054810    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:35.103950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:35.103911    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 20:11:35.104235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:35.104184    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 20:11:35.497697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:35.497616    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:35.804277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:35.803732    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:11:35.815824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:35.815785    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf" event={"ID":"9d356215-1ebc-4bd1-85ed-16245f324280","Type":"ContainerStarted","Data":"82d9b96ab0e52396b51584df790a42d767375b17a8fc6f044855690ae3196748"}
Nov 03 20:11:35.832900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:35.832848    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf" podStartSLOduration=2.832832946 podStartE2EDuration="2.832832946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:35.830590843 +0000 UTC m=+1888.499792314" watchObservedRunningTime="2026-11-03 20:11:35.832832946 +0000 UTC m=+1888.502034416"
Nov 03 20:11:36.160272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:36.138158    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 20:11:36.160272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:36.138235    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 20:11:36.498626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:36.485073    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:36.853692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:36.853601    5282 generic.go:356] "Generic (PLEG): container finished" podID="9d356215-1ebc-4bd1-85ed-16245f324280" containerID="82d9b96ab0e52396b51584df790a42d767375b17a8fc6f044855690ae3196748" exitCode=0
Nov 03 20:11:36.853868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:36.853807    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf" event={"ID":"9d356215-1ebc-4bd1-85ed-16245f324280","Type":"ContainerDied","Data":"82d9b96ab0e52396b51584df790a42d767375b17a8fc6f044855690ae3196748"}
Nov 03 20:11:37.102762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:37.102717    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 20:11:37.102955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:37.102794    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 20:11:37.518223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:37.508885    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:38.110321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.106823    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 03 20:11:38.110321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.106904    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 03 20:11:38.126130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.125638    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:38.149449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.149417    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:38.414252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.414146    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-openshift-ca\") pod \"9d356215-1ebc-4bd1-85ed-16245f324280\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") "
Nov 03 20:11:38.414500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.414478    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d356215-1ebc-4bd1-85ed-16245f324280-kube-api-access-kp9gz\" (UniqueName: \"kubernetes.io/projected/9d356215-1ebc-4bd1-85ed-16245f324280-kube-api-access-kp9gz\") pod \"9d356215-1ebc-4bd1-85ed-16245f324280\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") "
Nov 03 20:11:38.414578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.414553    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-xks-ca\") pod \"9d356215-1ebc-4bd1-85ed-16245f324280\" (UID: \"9d356215-1ebc-4bd1-85ed-16245f324280\") "
Nov 03 20:11:38.414744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.414713    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-openshift-ca" pod "9d356215-1ebc-4bd1-85ed-16245f324280" (UID: "9d356215-1ebc-4bd1-85ed-16245f324280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:38.415305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.415282    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:38.418077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.418046    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d356215-1ebc-4bd1-85ed-16245f324280-kube-api-access-kp9gz" pod "9d356215-1ebc-4bd1-85ed-16245f324280" (UID: "9d356215-1ebc-4bd1-85ed-16245f324280"). InnerVolumeSpecName "kube-api-access-kp9gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:38.418320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.418298    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-xks-ca" pod "9d356215-1ebc-4bd1-85ed-16245f324280" (UID: "9d356215-1ebc-4bd1-85ed-16245f324280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:38.517467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.517406    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp9gz\" (UniqueName: \"kubernetes.io/projected/9d356215-1ebc-4bd1-85ed-16245f324280-kube-api-access-kp9gz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:38.517467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.517440    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d356215-1ebc-4bd1-85ed-16245f324280-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:38.876137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.876104    5282 generic.go:356] "Generic (PLEG): container finished" podID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerID="124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4" exitCode=0
Nov 03 20:11:38.876786 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.876189    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" event={"ID":"71bb8111-a2fd-4e84-a7da-e63466d8b6d9","Type":"ContainerDied","Data":"124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4"}
Nov 03 20:11:38.876786 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.876773    5282 scope.go:122] "RemoveContainer" containerID="124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4"
Nov 03 20:11:38.878214 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.878051    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf" event={"ID":"9d356215-1ebc-4bd1-85ed-16245f324280","Type":"ContainerDied","Data":"bf3e93cf0e8fa56fccbc30673733850db0b77bc25cbd8a86672eea7ee8882c1f"}
Nov 03 20:11:38.878214 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.878109    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf3e93cf0e8fa56fccbc30673733850db0b77bc25cbd8a86672eea7ee8882c1f"
Nov 03 20:11:38.878214 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.878062    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"
Nov 03 20:11:38.879098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.879068    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:38.879219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.879143    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:38.879219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.879177    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:38.929123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:38.929090    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:39.117299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:39.116406    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:11:39.120568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:39.120536    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:11:39.458534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:39.458435    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:39.627342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:39.627211    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:39.916175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:39.916140    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" event={"ID":"71bb8111-a2fd-4e84-a7da-e63466d8b6d9","Type":"ContainerStarted","Data":"8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0"}
Nov 03 20:11:40.098478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.098429    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:11:40.099341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.099314    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:40.099498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.099357    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:40.099714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.099691    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:40.294479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.294426    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:40.781516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.781424    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:40.938586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.938526    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:40.940762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:40.940737    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:11:41.145292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:41.099566    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:41.145292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:41.099627    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:41.449564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:41.422827    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:42.098112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:42.098080    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:42.103231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:42.098680    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:42.556960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:42.556923    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:43.102224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:43.097173    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:43.102224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:43.097275    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:43.298903 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:43.298870    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:43.521229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:43.508876    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:43.779276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:43.769773    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:44.123277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:44.104681    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:44.123277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:44.104759    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:44.164287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:44.162664    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:44.436286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:44.407589    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"]
Nov 03 20:11:44.436574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:44.436545    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"]
Nov 03 20:11:44.465842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:44.420859    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:44.685248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:44.673112    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:45.125221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.124361    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:45.125221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.124425    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:45.628990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.628951    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:45.632277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.632240    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"]
Nov 03 20:11:45.633106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.633089    5282 cpu_manager.go:414] "containerMap: removing container" podUID="9d356215-1ebc-4bd1-85ed-16245f324280" containerName="revoke-keys"
Nov 03 20:11:45.633153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.633108    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d356215-1ebc-4bd1-85ed-16245f324280" containerName="revoke-keys"
Nov 03 20:11:45.633367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.633354    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d356215-1ebc-4bd1-85ed-16245f324280" containerName="revoke-keys"
Nov 03 20:11:45.637493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.637475    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:45.682638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.677509    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-caa01a\""
Nov 03 20:11:45.710131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.710088    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk247\" (UniqueName: \"kubernetes.io/projected/61953a05-7db9-4b00-97ce-1cb92090b97b-kube-api-access-sk247\") pod \"payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:45.715388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.710254    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61953a05-7db9-4b00-97ce-1cb92090b97b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:45.715388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.711730    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"]
Nov 03 20:11:45.716751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.716726    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"]
Nov 03 20:11:45.730758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.730724    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"]
Nov 03 20:11:45.730931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.730913    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:45.850303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.821785    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:45.873225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.861369    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61953a05-7db9-4b00-97ce-1cb92090b97b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:45.873225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.861433    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:45.873225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.861493    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzznq\" (UniqueName: \"kubernetes.io/projected/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-kube-api-access-fzznq\") pod \"payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:45.873225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.861822    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk247\" (UniqueName: \"kubernetes.io/projected/61953a05-7db9-4b00-97ce-1cb92090b97b-kube-api-access-sk247\") pod \"payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:45.873225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.863325    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61953a05-7db9-4b00-97ce-1cb92090b97b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:45.975500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:45.967722    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-caa01a-dockercfg-k64lm\""
Nov 03 20:11:46.123217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.107686    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:46.123217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.107751    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:46.167219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.161070    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk247\" (UniqueName: \"kubernetes.io/projected/61953a05-7db9-4b00-97ce-1cb92090b97b-kube-api-access-sk247\") pod \"payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") " pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:46.175223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.168407    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:46.175223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.168495    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzznq\" (UniqueName: \"kubernetes.io/projected/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-kube-api-access-fzznq\") pod \"payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:46.176519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.176469    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:46.267218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.255054    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzznq\" (UniqueName: \"kubernetes.io/projected/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-kube-api-access-fzznq\") pod \"payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:46.314709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.314667    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:46.354230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.344765    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:46.697757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.697477    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:46.923160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.923127    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"]
Nov 03 20:11:46.923381 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:11:46.923352    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bb5035_4ea1_4fe1_82e7_f8eea347bbb1.slice/crio-9db9b7f9f7a38397434642a4692408c5cd6bb35d536ad6294bbf3b9dfe51da91 WatchSource:0}: Error finding container 9db9b7f9f7a38397434642a4692408c5cd6bb35d536ad6294bbf3b9dfe51da91: Status 404 returned error can't find the container with id 9db9b7f9f7a38397434642a4692408c5cd6bb35d536ad6294bbf3b9dfe51da91
Nov 03 20:11:46.925519 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:11:46.925489    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61953a05_7db9_4b00_97ce_1cb92090b97b.slice/crio-17838213f07f4c4391bfb13b8fe6a3eae43f3f4b867bf688bc84d73d56324804 WatchSource:0}: Error finding container 17838213f07f4c4391bfb13b8fe6a3eae43f3f4b867bf688bc84d73d56324804: Status 404 returned error can't find the container with id 17838213f07f4c4391bfb13b8fe6a3eae43f3f4b867bf688bc84d73d56324804
Nov 03 20:11:46.928313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:46.928290    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"]
Nov 03 20:11:47.090017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.089985    5282 patch_prober.go:29] interesting pod/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 03 20:11:47.090246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.090037    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 03 20:11:47.317938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.314996    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:47.441223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.439224    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x" event={"ID":"61953a05-7db9-4b00-97ce-1cb92090b97b","Type":"ContainerStarted","Data":"dcf4e38093fd7b20c03923fd666c1bb83556060d4f60d9945c577df8eae6d818"}
Nov 03 20:11:47.441223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.439285    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x" event={"ID":"61953a05-7db9-4b00-97ce-1cb92090b97b","Type":"ContainerStarted","Data":"17838213f07f4c4391bfb13b8fe6a3eae43f3f4b867bf688bc84d73d56324804"}
Nov 03 20:11:47.441223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.440560    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:47.494221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.493528    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg" event={"ID":"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1","Type":"ContainerStarted","Data":"f7668fe421c3cb4d688f365ba8a6d65835e5267313cfc7d104d438117e41b8a5"}
Nov 03 20:11:47.495345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.493598    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:47.495345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.494921    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg" event={"ID":"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1","Type":"ContainerStarted","Data":"9db9b7f9f7a38397434642a4692408c5cd6bb35d536ad6294bbf3b9dfe51da91"}
Nov 03 20:11:47.509468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.509397    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x" podStartSLOduration=2.509379457 podStartE2EDuration="2.509379457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:47.487772501 +0000 UTC m=+1900.156973972" watchObservedRunningTime="2026-11-03 20:11:47.509379457 +0000 UTC m=+1900.178580927"
Nov 03 20:11:47.513607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:47.512637    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg" podStartSLOduration=2.5126212 podStartE2EDuration="2.5126212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:47.507361497 +0000 UTC m=+1900.176562969" watchObservedRunningTime="2026-11-03 20:11:47.5126212 +0000 UTC m=+1900.181822671"
Nov 03 20:11:48.136313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:48.131525    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:11:48.144885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:48.143920    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:11:48.176215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:48.167710    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:11:48.295218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:48.290772    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:49.098050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.097967    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:49.265438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.265393    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:49.437310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.437205    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr" podUID="b734719e-2eb8-40c7-8339-47831a5ce3ad" containerName="payload-pre-processing" containerID="cri-o://44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be" gracePeriod=25
Nov 03 20:11:49.446888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.446837    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd" podUID="21072de7-af50-4445-bc9b-39f704e02502" containerName="payload-processing" containerID="cri-o://91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6" gracePeriod=25
Nov 03 20:11:49.706700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.706670    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:49.770108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.770079    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:11:49.887106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.887055    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:49.902982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.895952    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:11:49.903289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.902733    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b734719e-2eb8-40c7-8339-47831a5ce3ad-kube-api-access-dwqtg\" (UniqueName: \"kubernetes.io/projected/b734719e-2eb8-40c7-8339-47831a5ce3ad-kube-api-access-dwqtg\") pod \"b734719e-2eb8-40c7-8339-47831a5ce3ad\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") "
Nov 03 20:11:49.903943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.903917    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b734719e-2eb8-40c7-8339-47831a5ce3ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b734719e-2eb8-40c7-8339-47831a5ce3ad-plugins-config-volume\") pod \"b734719e-2eb8-40c7-8339-47831a5ce3ad\" (UID: \"b734719e-2eb8-40c7-8339-47831a5ce3ad\") "
Nov 03 20:11:49.905057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.904485    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b734719e-2eb8-40c7-8339-47831a5ce3ad-plugins-config-volume" pod "b734719e-2eb8-40c7-8339-47831a5ce3ad" (UID: "b734719e-2eb8-40c7-8339-47831a5ce3ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:49.907953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.907928    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b734719e-2eb8-40c7-8339-47831a5ce3ad-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:49.960069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:49.959949    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b734719e-2eb8-40c7-8339-47831a5ce3ad-kube-api-access-dwqtg" pod "b734719e-2eb8-40c7-8339-47831a5ce3ad" (UID: "b734719e-2eb8-40c7-8339-47831a5ce3ad"). InnerVolumeSpecName "kube-api-access-dwqtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:50.056900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.056859    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21072de7-af50-4445-bc9b-39f704e02502-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21072de7-af50-4445-bc9b-39f704e02502-plugins-config-volume\") pod \"21072de7-af50-4445-bc9b-39f704e02502\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") "
Nov 03 20:11:50.057314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.057289    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21072de7-af50-4445-bc9b-39f704e02502-kube-api-access-b67ss\" (UniqueName: \"kubernetes.io/projected/21072de7-af50-4445-bc9b-39f704e02502-kube-api-access-b67ss\") pod \"21072de7-af50-4445-bc9b-39f704e02502\" (UID: \"21072de7-af50-4445-bc9b-39f704e02502\") "
Nov 03 20:11:50.058067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.058047    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqtg\" (UniqueName: \"kubernetes.io/projected/b734719e-2eb8-40c7-8339-47831a5ce3ad-kube-api-access-dwqtg\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:50.058727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.058706    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21072de7-af50-4445-bc9b-39f704e02502-plugins-config-volume" pod "21072de7-af50-4445-bc9b-39f704e02502" (UID: "21072de7-af50-4445-bc9b-39f704e02502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:50.065770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.065721    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21072de7-af50-4445-bc9b-39f704e02502-kube-api-access-b67ss" pod "21072de7-af50-4445-bc9b-39f704e02502" (UID: "21072de7-af50-4445-bc9b-39f704e02502"). InnerVolumeSpecName "kube-api-access-b67ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:50.069621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.069594    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"]
Nov 03 20:11:50.070765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.070744    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b734719e-2eb8-40c7-8339-47831a5ce3ad" containerName="payload-pre-processing"
Nov 03 20:11:50.070842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.070766    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b734719e-2eb8-40c7-8339-47831a5ce3ad" containerName="payload-pre-processing"
Nov 03 20:11:50.070842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.070807    5282 cpu_manager.go:414] "containerMap: removing container" podUID="21072de7-af50-4445-bc9b-39f704e02502" containerName="payload-processing"
Nov 03 20:11:50.070842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.070812    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21072de7-af50-4445-bc9b-39f704e02502" containerName="payload-processing"
Nov 03 20:11:50.070967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.070958    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b734719e-2eb8-40c7-8339-47831a5ce3ad" containerName="payload-pre-processing"
Nov 03 20:11:50.071001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.070974    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="21072de7-af50-4445-bc9b-39f704e02502" containerName="payload-processing"
Nov 03 20:11:50.075091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.075073    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.082026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.081993    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-facc79-openshift-default-dockercfg-fjlx2\""
Nov 03 20:11:50.093288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.093264    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"]
Nov 03 20:11:50.147535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.147499    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:50.159879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.159835    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-credential-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160017    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-podinfo\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160058    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-envoy\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160098    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-data\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160254    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-certs\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160335    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-token\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160360    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42d2e4-8389-474b-9c77-eb1fd340edad-istiod-ca-cert\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160401    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97x5\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-kube-api-access-v97x5\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.160712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.160691    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.161050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.161009    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b67ss\" (UniqueName: \"kubernetes.io/projected/21072de7-af50-4445-bc9b-39f704e02502-kube-api-access-b67ss\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:50.161050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.161037    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21072de7-af50-4445-bc9b-39f704e02502-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.305982    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306046    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-credential-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306171    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-podinfo\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306228    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-envoy\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306267    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-data\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306335    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-certs\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306392    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-token\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306412    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42d2e4-8389-474b-9c77-eb1fd340edad-istiod-ca-cert\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.306459    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97x5\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-kube-api-access-v97x5\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.307403    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.307642    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-credential-socket\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.311271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.308782    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-certs\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.352224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.312365    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-data\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.352224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.313035    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42d2e4-8389-474b-9c77-eb1fd340edad-istiod-ca-cert\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.374222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.363844    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:50.459226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.424513    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-podinfo\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.522592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.512974    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-token\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.567349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.563720    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97x5\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-kube-api-access-v97x5\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.677427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.654554    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-envoy\") pod \"e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") " pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.677427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.670611    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr" podUID="b734719e-2eb8-40c7-8339-47831a5ce3ad" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.111:50052\" within 1s: context deadline exceeded"
Nov 03 20:11:50.677427 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:11:50.670970    5282 logging.go:55] [core] [Channel #2156 SubChannel #2157]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.111:50052", ServerName: "10.133.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.111:50052: operation was canceled"
Nov 03 20:11:50.716663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.714534    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:50.753388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.733278    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:50.878560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.878454    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:50.891289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.891251    5282 generic.go:356] "Generic (PLEG): container finished" podID="b734719e-2eb8-40c7-8339-47831a5ce3ad" containerID="44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be" exitCode=0
Nov 03 20:11:50.897546 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.891639    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr" event={"ID":"b734719e-2eb8-40c7-8339-47831a5ce3ad","Type":"ContainerDied","Data":"44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be"}
Nov 03 20:11:50.897839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.897814    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr" event={"ID":"b734719e-2eb8-40c7-8339-47831a5ce3ad","Type":"ContainerDied","Data":"d34bf394caa8431183a171999f03adac63cedf98d5d8c5b3c291e264ff88c968"}
Nov 03 20:11:50.897985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.897969    5282 scope.go:122] "RemoveContainer" containerID="44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be"
Nov 03 20:11:50.898212 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:50.891758    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"
Nov 03 20:11:51.096788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.079587    5282 generic.go:356] "Generic (PLEG): container finished" podID="21072de7-af50-4445-bc9b-39f704e02502" containerID="91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6" exitCode=0
Nov 03 20:11:51.114112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.097326    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd" event={"ID":"21072de7-af50-4445-bc9b-39f704e02502","Type":"ContainerDied","Data":"91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6"}
Nov 03 20:11:51.114443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.114417    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd" event={"ID":"21072de7-af50-4445-bc9b-39f704e02502","Type":"ContainerDied","Data":"ca75aeeb05927a79fedca8218abef4374f2c5181fbae308176196e7ff6bb406f"}
Nov 03 20:11:51.114572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.113918    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:51.114726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.097617    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"
Nov 03 20:11:51.264603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.248808    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"]
Nov 03 20:11:51.264603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.249120    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" containerID="cri-o://2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e" gracePeriod=30
Nov 03 20:11:51.327651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.327620    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:51.327986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.327958    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"]
Nov 03 20:11:51.328500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.328470    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy" containerID="cri-o://90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1" gracePeriod=30
Nov 03 20:11:51.672023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.597123    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:51.689808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.684979    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:51.762294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.752407    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:51.850192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.850140    5282 scope.go:122] "RemoveContainer" containerID="44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be"
Nov 03 20:11:51.874338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.874289    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:51.879990 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:51.879929    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be\": container with ID starting with 44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be not found: ID does not exist" containerID="44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be"
Nov 03 20:11:51.880263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.880225    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be"} err="failed to get container status \"44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be\": rpc error: code = NotFound desc = could not find container \"44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be\": container with ID starting with 44922994eb5890bb75e5cc40b4a5c33e62bd8e616d3ae8b4596838c3d03887be not found: ID does not exist"
Nov 03 20:11:51.880524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.880505    5282 scope.go:122] "RemoveContainer" containerID="91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6"
Nov 03 20:11:51.881148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.881126    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:51.881884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:51.881837    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:11:52.047865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.047834    5282 scope.go:122] "RemoveContainer" containerID="91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6"
Nov 03 20:11:52.051899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.051866    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"]
Nov 03 20:11:52.056829 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:52.056640    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6\": container with ID starting with 91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6 not found: ID does not exist" containerID="91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6"
Nov 03 20:11:52.056829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.056673    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6"} err="failed to get container status \"91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6\": rpc error: code = NotFound desc = could not find container \"91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6\": container with ID starting with 91152103ab8b6bc0f27a806dced0ea80b96ddafee3c92ef4f4579ee87fbef2c6 not found: ID does not exist"
Nov 03 20:11:52.059457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.059397    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cd7f5c-57976b7948-scqtd"]
Nov 03 20:11:52.065265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.065245    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"]
Nov 03 20:11:52.069548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.069528    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cd7f5c-844f7f5d55-jxjkr"]
Nov 03 20:11:52.161265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.161236    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"]
Nov 03 20:11:52.161979 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:11:52.161942    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f42d2e4_8389_474b_9c77_eb1fd340edad.slice/crio-ebca3f4007f54d340897fe6f339659d7e746377e912802e71956366495f7ec52 WatchSource:0}: Error finding container ebca3f4007f54d340897fe6f339659d7e746377e912802e71956366495f7ec52: Status 404 returned error can't find the container with id ebca3f4007f54d340897fe6f339659d7e746377e912802e71956366495f7ec52
Nov 03 20:11:52.163252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.163225    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:52.164775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.164742    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:52.164854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.164806    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:52.164854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.164835    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:11:52.302275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.299542    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:52.788360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:52.788328    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:53.145255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.145158    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" event={"ID":"0f42d2e4-8389-474b-9c77-eb1fd340edad","Type":"ContainerStarted","Data":"d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38"}
Nov 03 20:11:53.145255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.145224    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" event={"ID":"0f42d2e4-8389-474b-9c77-eb1fd340edad","Type":"ContainerStarted","Data":"ebca3f4007f54d340897fe6f339659d7e746377e912802e71956366495f7ec52"}
Nov 03 20:11:53.173547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.173500    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podStartSLOduration=3.173487516 podStartE2EDuration="3.173487516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:53.167780378 +0000 UTC m=+1905.836981862" watchObservedRunningTime="2026-11-03 20:11:53.173487516 +0000 UTC m=+1905.842689052"
Nov 03 20:11:53.495919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.495875    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21072de7-af50-4445-bc9b-39f704e02502" path="/var/lib/kubelet/pods/21072de7-af50-4445-bc9b-39f704e02502/volumes"
Nov 03 20:11:53.496689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.496667    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b734719e-2eb8-40c7-8339-47831a5ce3ad" path="/var/lib/kubelet/pods/b734719e-2eb8-40c7-8339-47831a5ce3ad/volumes"
Nov 03 20:11:53.588024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.587993    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"]
Nov 03 20:11:53.593681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.593660    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.597329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.597307    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:11:53.597665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.597634    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:11:53.600771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.600749    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"]
Nov 03 20:11:53.649283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.649252    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.649470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.649291    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.649642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.649618    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbr84\" (UniqueName: \"kubernetes.io/projected/d28ea5ce-1fd2-4252-988b-7754c0442282-kube-api-access-nbr84\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.693099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.693070    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:53.786698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.786569    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbr84\" (UniqueName: \"kubernetes.io/projected/d28ea5ce-1fd2-4252-988b-7754c0442282-kube-api-access-nbr84\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.787424 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.787182    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.787609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.787593    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.793251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.789323    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.793717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.793613    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:53.827143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.822004    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbr84\" (UniqueName: \"kubernetes.io/projected/d28ea5ce-1fd2-4252-988b-7754c0442282-kube-api-access-nbr84\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.835967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.834111    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:53.851005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.850864    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:53.916837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:53.916801    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:54.119761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:54.119676    5282 patch_prober.go:29] interesting pod/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:11:54.119761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:54.119740    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:11:54.154451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:54.154413    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:54.155835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:54.155813    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:11:54.209333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:54.209296    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"]
Nov 03 20:11:54.674100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:54.674008    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:55.165184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:55.165149    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v" event={"ID":"d28ea5ce-1fd2-4252-988b-7754c0442282","Type":"ContainerStarted","Data":"afa7739934c389042c9171d889ec978c83d00c9b59de3bd7e9995f004a7801aa"}
Nov 03 20:11:55.165391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:55.165277    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v" event={"ID":"d28ea5ce-1fd2-4252-988b-7754c0442282","Type":"ContainerStarted","Data":"643ffedcb23a28a901c1392d4645d1d3a85c0a2ec0543b612f5aa9e0fce78936"}
Nov 03 20:11:55.187602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:55.187558    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v" podStartSLOduration=2.187545001 podStartE2EDuration="2.187545001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:55.182410641 +0000 UTC m=+1907.851612110" watchObservedRunningTime="2026-11-03 20:11:55.187545001 +0000 UTC m=+1907.856746471"
Nov 03 20:11:55.622596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:55.622560    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:55.822318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:55.822281    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:56.218221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.217309    5282 generic.go:356] "Generic (PLEG): container finished" podID="d28ea5ce-1fd2-4252-988b-7754c0442282" containerID="afa7739934c389042c9171d889ec978c83d00c9b59de3bd7e9995f004a7801aa" exitCode=0
Nov 03 20:11:56.221600 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.220708    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v" event={"ID":"d28ea5ce-1fd2-4252-988b-7754c0442282","Type":"ContainerDied","Data":"afa7739934c389042c9171d889ec978c83d00c9b59de3bd7e9995f004a7801aa"}
Nov 03 20:11:56.708592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.708566    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:56.744663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.744591    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:11:56.860691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.860656    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:56.890278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890245    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-data\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.890467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890299    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-credential-socket\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.890467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890356    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-socket\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.890467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890416    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istiod-ca-cert\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.890632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890502    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-kube-api-access-mzlj8\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-kube-api-access-mzlj8\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.890632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890531    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-data" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.890632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890545    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-token\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-token\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.890632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890544    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-credential-socket" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.890851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890641    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-envoy\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.890851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890681    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-socket" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.890851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890743    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-envoy\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.890851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890796    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-socket\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.891059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890865    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istiod-ca-cert" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:56.891059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890906    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-token\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-token\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.891059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.890985    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-kube-api-access-8h7xm\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-kube-api-access-8h7xm\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.891259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891108    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-socket" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.891259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891023    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-credential-socket\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.891376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891324    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-certs\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.891376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891368    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-podinfo\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.891474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891421    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-certs\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.891474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891437    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-credential-socket" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.891587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891470    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-data\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.891587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891540    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-podinfo\") pod \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\" (UID: \"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9\") "
Nov 03 20:11:56.891587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891578    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-certs" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.891750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891621    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27507f86-3733-48d7-9223-c5fb07e94f9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27507f86-3733-48d7-9223-c5fb07e94f9e-istiod-ca-cert\") pod \"27507f86-3733-48d7-9223-c5fb07e94f9e\" (UID: \"27507f86-3733-48d7-9223-c5fb07e94f9e\") "
Nov 03 20:11:56.891814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.891761    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-data" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.892084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892060    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-certs" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.892176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892145    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27507f86-3733-48d7-9223-c5fb07e94f9e-istiod-ca-cert" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:56.892869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892841    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.892986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892922    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.892986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892939    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.892986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892954    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27507f86-3733-48d7-9223-c5fb07e94f9e-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.892986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892967    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.892986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892982    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.893303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.892996    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.893303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.893013    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.893303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.893026    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.893303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.893040    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:56.893500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.893425    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-token" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:56.893500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.893477    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-kube-api-access-8h7xm" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "kube-api-access-8h7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:56.893862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.893832    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-envoy" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.893966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.893924    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-kube-api-access-mzlj8" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "kube-api-access-mzlj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:56.894228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.894172    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-podinfo" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:56.894320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.894293    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-token" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:56.894547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.894529    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-envoy" pod "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" (UID: "4f8fc6b5-48c4-493d-ac3d-7c7913d325a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:56.895074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:56.895056    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-podinfo" pod "27507f86-3733-48d7-9223-c5fb07e94f9e" (UID: "27507f86-3733-48d7-9223-c5fb07e94f9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:57.088452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.059509    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:57.110135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110038    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7xm\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-kube-api-access-8h7xm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.110135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110087    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.110135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110101    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.110528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110114    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzlj8\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-kube-api-access-mzlj8\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.110691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110642    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.110852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110667    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27507f86-3733-48d7-9223-c5fb07e94f9e-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.110972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110930    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.110972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.110953    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.483630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.479626    5282 generic.go:356] "Generic (PLEG): container finished" podID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerID="90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1" exitCode=0
Nov 03 20:11:57.498142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.498076    5282 generic.go:356] "Generic (PLEG): container finished" podID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerID="2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e" exitCode=0
Nov 03 20:11:57.551566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.551531    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"
Nov 03 20:11:57.553899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.553502    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" event={"ID":"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9","Type":"ContainerDied","Data":"90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1"}
Nov 03 20:11:57.553899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.553557    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq" event={"ID":"4f8fc6b5-48c4-493d-ac3d-7c7913d325a9","Type":"ContainerDied","Data":"1eb81fe044b32304ab54f1b35200424c76660143682a9c0bf08096d8eba0fc4b"}
Nov 03 20:11:57.553899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.553588    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" event={"ID":"27507f86-3733-48d7-9223-c5fb07e94f9e","Type":"ContainerDied","Data":"2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e"}
Nov 03 20:11:57.553899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.553617    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc" event={"ID":"27507f86-3733-48d7-9223-c5fb07e94f9e","Type":"ContainerDied","Data":"6695b9bac2c95805b81b49c18d7c4d315848e56de23aea3666debb7a9495b119"}
Nov 03 20:11:57.553899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.553659    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"
Nov 03 20:11:57.554512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.554455    5282 scope.go:122] "RemoveContainer" containerID="90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1"
Nov 03 20:11:57.587612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.587584    5282 scope.go:122] "RemoveContainer" containerID="90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1"
Nov 03 20:11:57.587934 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:57.587903    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1\": container with ID starting with 90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1 not found: ID does not exist" containerID="90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1"
Nov 03 20:11:57.588032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.587949    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1"} err="failed to get container status \"90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1\": rpc error: code = NotFound desc = could not find container \"90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1\": container with ID starting with 90388efc6d39ff2f7dc76234aab1274d39633d34639bfc0778cb7222ab6015e1 not found: ID does not exist"
Nov 03 20:11:57.588032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.587972    5282 scope.go:122] "RemoveContainer" containerID="2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e"
Nov 03 20:11:57.601534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.601513    5282 scope.go:122] "RemoveContainer" containerID="da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b"
Nov 03 20:11:57.611406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.607655    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"]
Nov 03 20:11:57.613825 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.613796    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-5kldq"]
Nov 03 20:11:57.622941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.622915    5282 scope.go:122] "RemoveContainer" containerID="2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e"
Nov 03 20:11:57.623355 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:57.623317    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e\": container with ID starting with 2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e not found: ID does not exist" containerID="2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e"
Nov 03 20:11:57.623472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.623354    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e"} err="failed to get container status \"2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e\": rpc error: code = NotFound desc = could not find container \"2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e\": container with ID starting with 2d59456ecf673919c3010d8a45886606fa7f4ba544308e28e3756d88be50849e not found: ID does not exist"
Nov 03 20:11:57.623472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.623380    5282 scope.go:122] "RemoveContainer" containerID="da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b"
Nov 03 20:11:57.624573 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:11:57.624545    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b\": container with ID starting with da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b not found: ID does not exist" containerID="da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b"
Nov 03 20:11:57.624755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.624578    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b"} err="failed to get container status \"da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b\": rpc error: code = NotFound desc = could not find container \"da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b\": container with ID starting with da52b42bd04a982bf66e89da67fb05bcfa09f94fdd979c7c55c1c59223dd675b not found: ID does not exist"
Nov 03 20:11:57.629051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.629023    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"]
Nov 03 20:11:57.635164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.635133    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b9dc48-openshift-default-75564dbdc5-6wsdc"]
Nov 03 20:11:57.765249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.757420    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:57.864840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.864810    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-xks-ca\") pod \"d28ea5ce-1fd2-4252-988b-7754c0442282\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") "
Nov 03 20:11:57.865267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.864852    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-openshift-ca\") pod \"d28ea5ce-1fd2-4252-988b-7754c0442282\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") "
Nov 03 20:11:57.865267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.865015    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28ea5ce-1fd2-4252-988b-7754c0442282-kube-api-access-nbr84\" (UniqueName: \"kubernetes.io/projected/d28ea5ce-1fd2-4252-988b-7754c0442282-kube-api-access-nbr84\") pod \"d28ea5ce-1fd2-4252-988b-7754c0442282\" (UID: \"d28ea5ce-1fd2-4252-988b-7754c0442282\") "
Nov 03 20:11:57.865267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.865249    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-openshift-ca" pod "d28ea5ce-1fd2-4252-988b-7754c0442282" (UID: "d28ea5ce-1fd2-4252-988b-7754c0442282"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:57.865835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.865813    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.867263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.867239    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-xks-ca" pod "d28ea5ce-1fd2-4252-988b-7754c0442282" (UID: "d28ea5ce-1fd2-4252-988b-7754c0442282"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:57.870804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.870775    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28ea5ce-1fd2-4252-988b-7754c0442282-kube-api-access-nbr84" pod "d28ea5ce-1fd2-4252-988b-7754c0442282" (UID: "d28ea5ce-1fd2-4252-988b-7754c0442282"). InnerVolumeSpecName "kube-api-access-nbr84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:57.878874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.878818    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:57.885888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.885864    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"]
Nov 03 20:11:57.886693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886677    5282 cpu_manager.go:414] "containerMap: removing container" podUID="d28ea5ce-1fd2-4252-988b-7754c0442282" containerName="revoke-keys"
Nov 03 20:11:57.886781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886693    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28ea5ce-1fd2-4252-988b-7754c0442282" containerName="revoke-keys"
Nov 03 20:11:57.886781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886733    5282 cpu_manager.go:414] "containerMap: removing container" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy"
Nov 03 20:11:57.886781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886738    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy"
Nov 03 20:11:57.886781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886761    5282 cpu_manager.go:414] "containerMap: removing container" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy"
Nov 03 20:11:57.886781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886766    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy"
Nov 03 20:11:57.886781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886781    5282 cpu_manager.go:414] "containerMap: removing container" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy"
Nov 03 20:11:57.887089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886786    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy"
Nov 03 20:11:57.887089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886899    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy"
Nov 03 20:11:57.887089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886909    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" containerName="istio-proxy"
Nov 03 20:11:57.887089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886917    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" containerName="istio-proxy"
Nov 03 20:11:57.887089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.886926    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28ea5ce-1fd2-4252-988b-7754c0442282" containerName="revoke-keys"
Nov 03 20:11:57.890930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.890911    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:57.895652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.895630    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-facc79-dockercfg-grffh\""
Nov 03 20:11:57.895758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.895654    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-facc79\""
Nov 03 20:11:57.901978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.901954    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"]
Nov 03 20:11:57.968961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.968922    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:57.969263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.969238    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfgd\" (UniqueName: \"kubernetes.io/projected/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-kube-api-access-6lfgd\") pod \"payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:57.969645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.969613    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d28ea5ce-1fd2-4252-988b-7754c0442282-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:57.969715 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:57.969649    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbr84\" (UniqueName: \"kubernetes.io/projected/d28ea5ce-1fd2-4252-988b-7754c0442282-kube-api-access-nbr84\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:58.013331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.013299    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:58.086222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.073777    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:58.086630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.086596    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfgd\" (UniqueName: \"kubernetes.io/projected/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-kube-api-access-6lfgd\") pod \"payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:58.110217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.089610    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:58.149276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.110889    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfgd\" (UniqueName: \"kubernetes.io/projected/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-kube-api-access-6lfgd\") pod \"payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:58.202772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.198442    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:58.265977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.265655    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:58.582236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.581918    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:11:58.582236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.582015    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v" event={"ID":"d28ea5ce-1fd2-4252-988b-7754c0442282","Type":"ContainerDied","Data":"643ffedcb23a28a901c1392d4645d1d3a85c0a2ec0543b612f5aa9e0fce78936"}
Nov 03 20:11:58.582236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.582054    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643ffedcb23a28a901c1392d4645d1d3a85c0a2ec0543b612f5aa9e0fce78936"
Nov 03 20:11:58.888235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.874763    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"]
Nov 03 20:11:58.921518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:58.921332    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:11:59.458419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.458338    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27507f86-3733-48d7-9223-c5fb07e94f9e" path="/var/lib/kubelet/pods/27507f86-3733-48d7-9223-c5fb07e94f9e/volumes"
Nov 03 20:11:59.458935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.458918    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8fc6b5-48c4-493d-ac3d-7c7913d325a9" path="/var/lib/kubelet/pods/4f8fc6b5-48c4-493d-ac3d-7c7913d325a9/volumes"
Nov 03 20:11:59.547276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.547231    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:11:59.547703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.547683    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:11:59.600790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.600739    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx" event={"ID":"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd","Type":"ContainerStarted","Data":"22e4d7c590f4c69a7c4d4907440f5f6bec1b053873947979ae8e76f69ffc7d03"}
Nov 03 20:11:59.600959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.600817    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx" event={"ID":"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd","Type":"ContainerStarted","Data":"b0bada3412eca1be643dd062e2de5bc42ad2662ddfd1e694108ab801f127afb0"}
Nov 03 20:11:59.601327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.601301    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:11:59.622992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:11:59.622939    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx" podStartSLOduration=2.622924834 podStartE2EDuration="2.622924834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:59.619967617 +0000 UTC m=+1912.289169086" watchObservedRunningTime="2026-11-03 20:11:59.622924834 +0000 UTC m=+1912.292126336"
Nov 03 20:12:00.029220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:00.025745    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:01.369114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:01.369068    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:01.710280 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:01.706054    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:12:01.919312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:01.919269    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"]
Nov 03 20:12:01.926486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:01.926163    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:01.930998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:01.930975    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-caa01a-serving-cert\""
Nov 03 20:12:01.935275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:01.935248    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"]
Nov 03 20:12:02.012238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.011605    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:02.038773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.037859    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-openshift-service-ca\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.038773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.037951    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-maas-api-tls\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.038773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.038162    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-metrics-certs\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.038773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.038301    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4z9f\" (UniqueName: \"kubernetes.io/projected/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-kube-api-access-j4z9f\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.143298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.136381    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:02.181222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.170527    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-openshift-service-ca\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.181222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.170646    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-maas-api-tls\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.181222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.170977    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-metrics-certs\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.181222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.171090    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4z9f\" (UniqueName: \"kubernetes.io/projected/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-kube-api-access-j4z9f\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.227880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.227741    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-openshift-service-ca\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.247219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.243274    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4z9f\" (UniqueName: \"kubernetes.io/projected/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-kube-api-access-j4z9f\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.370292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.367266    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-metrics-certs\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.395047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.395007    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-maas-api-tls\") pod \"maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.563346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.563310    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:02.880139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:02.874731    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:03.610774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:03.610734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:03.649427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:03.622723    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"]
Nov 03 20:12:03.722222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:03.719518    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"]
Nov 03 20:12:03.795009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:03.786080    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:03.984724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:03.984670    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:04.014596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:04.014563    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz" event={"ID":"09a6dcfe-9e9a-47ab-aa1d-7f295178c637","Type":"ContainerStarted","Data":"7721f31622150f2332402a194ebf04107c7aa28bdc7b0403ab7928a0f4db64c8"}
Nov 03 20:12:04.508221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:04.504474    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:04.723669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:04.723627    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:12:04.746592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:04.745480    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:12:05.025805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:05.025769    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz" event={"ID":"09a6dcfe-9e9a-47ab-aa1d-7f295178c637","Type":"ContainerStarted","Data":"e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4"}
Nov 03 20:12:05.025986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:05.025966    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:05.045497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:05.045448    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz" podStartSLOduration=3.414338664 podStartE2EDuration="4.045435162s" totalImagesPullingTime="631.096498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:05.043856939 +0000 UTC m=+1917.713058411" watchObservedRunningTime="2026-11-03 20:12:05.045435162 +0000 UTC m=+1917.714636632"
Nov 03 20:12:05.117889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:05.117855    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:05.899732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:05.899698    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:06.561897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:06.561828    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:07.198281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:07.198239    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:07.885714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:07.885680    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:08.632992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:08.632907    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:08.722312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:08.722267    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc" podUID="7252bd12-b6ea-4a9b-b60b-f53a705a641d" containerName="payload-processing" containerID="cri-o://14daa48c7d1d6af450d6433427be1c44d4272368d233f1d8bfa1b39e5ff03948" gracePeriod=25
Nov 03 20:12:08.843251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:08.840927    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"]
Nov 03 20:12:08.843251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:08.841224    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy" containerID="cri-o://693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b" gracePeriod=30
Nov 03 20:12:08.848346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:08.848319    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"]
Nov 03 20:12:08.848558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:08.848536    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy" containerID="cri-o://13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05" gracePeriod=30
Nov 03 20:12:09.094324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.094296    5282 generic.go:356] "Generic (PLEG): container finished" podID="7252bd12-b6ea-4a9b-b60b-f53a705a641d" containerID="14daa48c7d1d6af450d6433427be1c44d4272368d233f1d8bfa1b39e5ff03948" exitCode=0
Nov 03 20:12:09.094555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.094364    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc" event={"ID":"7252bd12-b6ea-4a9b-b60b-f53a705a641d","Type":"ContainerDied","Data":"14daa48c7d1d6af450d6433427be1c44d4272368d233f1d8bfa1b39e5ff03948"}
Nov 03 20:12:09.094705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.094684    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc" event={"ID":"7252bd12-b6ea-4a9b-b60b-f53a705a641d","Type":"ContainerDied","Data":"7784333f7291ab91469e94f435f38646d37b217bd8b97b31b03d82aa4cc81092"}
Nov 03 20:12:09.094773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.094718    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7784333f7291ab91469e94f435f38646d37b217bd8b97b31b03d82aa4cc81092"
Nov 03 20:12:09.100731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.100709    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:12:09.200104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.200076    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7252bd12-b6ea-4a9b-b60b-f53a705a641d-kube-api-access-rl2wt\" (UniqueName: \"kubernetes.io/projected/7252bd12-b6ea-4a9b-b60b-f53a705a641d-kube-api-access-rl2wt\") pod \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") "
Nov 03 20:12:09.200297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.200192    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7252bd12-b6ea-4a9b-b60b-f53a705a641d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7252bd12-b6ea-4a9b-b60b-f53a705a641d-plugins-config-volume\") pod \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\" (UID: \"7252bd12-b6ea-4a9b-b60b-f53a705a641d\") "
Nov 03 20:12:09.200607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.200578    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7252bd12-b6ea-4a9b-b60b-f53a705a641d-plugins-config-volume" pod "7252bd12-b6ea-4a9b-b60b-f53a705a641d" (UID: "7252bd12-b6ea-4a9b-b60b-f53a705a641d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:09.201100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.201077    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7252bd12-b6ea-4a9b-b60b-f53a705a641d-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:09.202318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.202291    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7252bd12-b6ea-4a9b-b60b-f53a705a641d-kube-api-access-rl2wt" pod "7252bd12-b6ea-4a9b-b60b-f53a705a641d" (UID: "7252bd12-b6ea-4a9b-b60b-f53a705a641d"). InnerVolumeSpecName "kube-api-access-rl2wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:09.303357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.303323    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2wt\" (UniqueName: \"kubernetes.io/projected/7252bd12-b6ea-4a9b-b60b-f53a705a641d-kube-api-access-rl2wt\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:09.367278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.365764    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:09.983490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:09.983450    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:10.127258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.127180    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"
Nov 03 20:12:10.180007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.179944    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"]
Nov 03 20:12:10.185938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.185894    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-acd98d21-784b68d996-kjpjc"]
Nov 03 20:12:10.539246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.539205    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"]
Nov 03 20:12:10.540577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.540553    5282 cpu_manager.go:414] "containerMap: removing container" podUID="7252bd12-b6ea-4a9b-b60b-f53a705a641d" containerName="payload-processing"
Nov 03 20:12:10.540577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.540574    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7252bd12-b6ea-4a9b-b60b-f53a705a641d" containerName="payload-processing"
Nov 03 20:12:10.540793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.540750    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7252bd12-b6ea-4a9b-b60b-f53a705a641d" containerName="payload-processing"
Nov 03 20:12:10.548385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.548358    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.555572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.555517    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-43b780-serving-cert\""
Nov 03 20:12:10.582151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.578319    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"]
Nov 03 20:12:10.627483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.627451    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:10.648240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.648169    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-maas-api-tls\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.648390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.648287    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-metrics-certs\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.648451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.648398    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thklf\" (UniqueName: \"kubernetes.io/projected/2662b650-8f0f-4178-9751-d210b42fabed-kube-api-access-thklf\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.648451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.648423    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2662b650-8f0f-4178-9751-d210b42fabed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.807273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.790892    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-maas-api-tls\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.807273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.790944    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-metrics-certs\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.807273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.791083    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thklf\" (UniqueName: \"kubernetes.io/projected/2662b650-8f0f-4178-9751-d210b42fabed-kube-api-access-thklf\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.807273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.791115    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2662b650-8f0f-4178-9751-d210b42fabed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.807273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.792489    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2662b650-8f0f-4178-9751-d210b42fabed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.807273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.803807    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:10.807273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.805284    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:12:10.840069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.840018    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-maas-api-tls\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.853669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.853638    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-metrics-certs\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:10.959066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:10.959027    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thklf\" (UniqueName: \"kubernetes.io/projected/2662b650-8f0f-4178-9751-d210b42fabed-kube-api-access-thklf\") pod \"maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:11.071095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:11.071018    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:12:11.186855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:11.186820    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:11.556675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:11.556635    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7252bd12-b6ea-4a9b-b60b-f53a705a641d" path="/var/lib/kubelet/pods/7252bd12-b6ea-4a9b-b60b-f53a705a641d/volumes"
Nov 03 20:12:11.626226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:11.618389    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"]
Nov 03 20:12:11.790584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:11.790518    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:11.917605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:11.917524    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:12.055557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.055522    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"]
Nov 03 20:12:12.083305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.070072    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:12.257267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.249891    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.343245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.319018    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:12:12.385237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.379067    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:12:12.385237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.381488    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.385237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.381536    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.385237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.381713    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhnl\" (UniqueName: \"kubernetes.io/projected/18ff6775-eb23-46df-ad3e-e361447f36d3-kube-api-access-glhnl\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.491431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.491396    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhnl\" (UniqueName: \"kubernetes.io/projected/18ff6775-eb23-46df-ad3e-e361447f36d3-kube-api-access-glhnl\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.491814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.491789    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.491896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.491840    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.492693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.492670    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.551565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.547102    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhnl\" (UniqueName: \"kubernetes.io/projected/18ff6775-eb23-46df-ad3e-e361447f36d3-kube-api-access-glhnl\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:12.757264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:12.748376    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9" event={"ID":"2662b650-8f0f-4178-9751-d210b42fabed","Type":"ContainerStarted","Data":"12360eac9c1def44c8c103e6f211b3e1d144774aeff2448b0471e23b7c86ca65"}
Nov 03 20:12:13.016680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.016642    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:13.100620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.100591    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"]
Nov 03 20:12:13.249391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.248959    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbe151ac-m2xl2\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:13.364432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.363482    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:13.713555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.713451    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:13.835965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.833633    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9" event={"ID":"2662b650-8f0f-4178-9751-d210b42fabed","Type":"ContainerStarted","Data":"5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8"}
Nov 03 20:12:13.835965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.835258    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:13.916450 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:13.916402    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ff6775_eb23_46df_ad3e_e361447f36d3.slice/crio-2a986692aad21440708eb921ee943f972c3db724763fc6e6c075969bfa8e72be WatchSource:0}: Error finding container 2a986692aad21440708eb921ee943f972c3db724763fc6e6c075969bfa8e72be: Status 404 returned error can't find the container with id 2a986692aad21440708eb921ee943f972c3db724763fc6e6c075969bfa8e72be
Nov 03 20:12:13.941056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.940998    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9" podStartSLOduration=3.176006653 podStartE2EDuration="3.940978251s" totalImagesPullingTime="764.971598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:13.917516248 +0000 UTC m=+1926.586717719" watchObservedRunningTime="2026-11-03 20:12:13.940978251 +0000 UTC m=+1926.610179726"
Nov 03 20:12:13.946402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:13.945412    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"]
Nov 03 20:12:14.368051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.368021    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:12:14.402630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.402598    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"]
Nov 03 20:12:14.403477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.403460    5282 cpu_manager.go:414] "containerMap: removing container" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy"
Nov 03 20:12:14.403477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.403476    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy"
Nov 03 20:12:14.403643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.403633    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerName="istio-proxy"
Nov 03 20:12:14.407868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.407842    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.412566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.412532    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-certs\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.412977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.412935    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-certs" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.413107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.412988    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-kube-api-access-2q8sb\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-kube-api-access-2q8sb\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.413107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.413102    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-data\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.413338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.413137    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-podinfo\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.413338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.413273    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-token\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-token\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.413519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.413397    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f4e2b4-d910-4268-a01a-ca6e785a6319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f4e2b4-d910-4268-a01a-ca6e785a6319-istiod-ca-cert\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.414864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.414835    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-socket\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.415042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.415022    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-envoy\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.415106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.415091    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-credential-socket\") pod \"77f4e2b4-d910-4268-a01a-ca6e785a6319\" (UID: \"77f4e2b4-d910-4268-a01a-ca6e785a6319\") "
Nov 03 20:12:14.419503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.419232    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.419969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.419670    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-credential-socket" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.422872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.422847    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-data" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.423313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.423278    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f4e2b4-d910-4268-a01a-ca6e785a6319-istiod-ca-cert" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:14.431366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.423550    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-acd98d21-serving-cert\""
Nov 03 20:12:14.431366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.424414    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-socket" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.437433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.437396    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-token" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:14.437813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.437793    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-envoy" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.438183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.438026    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-kube-api-access-2q8sb" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "kube-api-access-2q8sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:14.438377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.438346    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-podinfo" pod "77f4e2b4-d910-4268-a01a-ca6e785a6319" (UID: "77f4e2b4-d910-4268-a01a-ca6e785a6319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:14.438958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.438886    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"]
Nov 03 20:12:14.520507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.520426    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.520507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.520473    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.520745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.520723    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:12:14.521006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.520979    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.521213 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521146    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nhb\" (UniqueName: \"kubernetes.io/projected/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-kube-api-access-w9nhb\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.521491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521466    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q8sb\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-kube-api-access-2q8sb\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.521596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521507    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.521596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521530    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.521596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521547    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.521596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521561    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77f4e2b4-d910-4268-a01a-ca6e785a6319-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.521596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521576    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.521596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521589    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.521791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.521604    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77f4e2b4-d910-4268-a01a-ca6e785a6319-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.556291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.556251    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"]
Nov 03 20:12:14.557140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.557125    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy"
Nov 03 20:12:14.557183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.557141    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy"
Nov 03 20:12:14.557340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.557327    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerName="istio-proxy"
Nov 03 20:12:14.561363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.561336    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.586730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.586484    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:14.589324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.587499    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"]
Nov 03 20:12:14.623058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623018    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-data\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623241    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istiod-ca-cert\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623279    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-podinfo\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623328    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-certs\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623356    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-socket\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623550    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-kube-api-access-xpftg\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-kube-api-access-xpftg\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623631    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-token\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-token\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623698    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-envoy\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.623774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.623747    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-credential-socket\") pod \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\" (UID: \"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc\") "
Nov 03 20:12:14.624288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624253    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.624386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624341    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.624386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624376    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.624509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624487    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.624595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624577    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.624684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624670    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.624748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624734    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nhb\" (UniqueName: \"kubernetes.io/projected/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-kube-api-access-w9nhb\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.624823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624809    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.624882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624860    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.624937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.624884    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.625030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.625015    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.625111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.625094    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.625172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.625126    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbw64\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-kube-api-access-xbw64\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.625873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.625845    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-credential-socket" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.625984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.625957    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-data" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.626719 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.626693    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-openshift-service-ca\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.628532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.628504    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-podinfo" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:14.628870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.628817    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istiod-ca-cert" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:14.629191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.629164    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-certs" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.629395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.629372    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-socket" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.632594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.632556    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-envoy" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:14.632693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.632592    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-token" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:14.632947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.632919    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-metrics-certs\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.633801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.633779    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-kube-api-access-xpftg" pod "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" (UID: "0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc"). InnerVolumeSpecName "kube-api-access-xpftg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:14.633929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.633906    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.639645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.639623    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nhb\" (UniqueName: \"kubernetes.io/projected/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-kube-api-access-w9nhb\") pod \"maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.763271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.749501    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.801376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.801282    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.801626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.801609    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.779545    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.801713    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbw64\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-kube-api-access-xbw64\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.801955    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802020    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802096    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802228    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802379    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.802617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802597    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.802752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802740    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.802830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802820    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.802901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802891    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.802975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.802965    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpftg\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-kube-api-access-xpftg\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.803042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.803033    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.803138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.803126    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.824341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.810331    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.824341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.810363    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:14.824341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.801682    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.824341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.804789    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.824341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.804580    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.824341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.803936    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.824341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.818896    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.825044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.824970    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.831354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.831020    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:14.843981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.843764    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbw64\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-kube-api-access-xbw64\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.844478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.844404    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.855819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.855714    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2" event={"ID":"18ff6775-eb23-46df-ad3e-e361447f36d3","Type":"ContainerStarted","Data":"4faf6b80264df00fbc8eb6992c4baebfaab3a835ae38f72d281e29bab6b5d6d3"}
Nov 03 20:12:14.855819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.855783    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2" event={"ID":"18ff6775-eb23-46df-ad3e-e361447f36d3","Type":"ContainerStarted","Data":"2a986692aad21440708eb921ee943f972c3db724763fc6e6c075969bfa8e72be"}
Nov 03 20:12:14.861083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.859952    5282 generic.go:356] "Generic (PLEG): container finished" podID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" containerID="13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05" exitCode=0
Nov 03 20:12:14.861083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.860084    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" event={"ID":"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc","Type":"ContainerDied","Data":"13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05"}
Nov 03 20:12:14.861083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.860129    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd" event={"ID":"0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc","Type":"ContainerDied","Data":"151cd2a803dd7dd096ba72af91992f22c588717c4f5c323a8b6dc4b09b5ddd60"}
Nov 03 20:12:14.861083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.860161    5282 scope.go:122] "RemoveContainer" containerID="13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05"
Nov 03 20:12:14.861083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.860345    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"
Nov 03 20:12:14.864241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.863332    5282 generic.go:356] "Generic (PLEG): container finished" podID="77f4e2b4-d910-4268-a01a-ca6e785a6319" containerID="693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b" exitCode=0
Nov 03 20:12:14.865242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.864590    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"
Nov 03 20:12:14.865563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.865536    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" event={"ID":"77f4e2b4-d910-4268-a01a-ca6e785a6319","Type":"ContainerDied","Data":"693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b"}
Nov 03 20:12:14.865661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.865588    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8" event={"ID":"77f4e2b4-d910-4268-a01a-ca6e785a6319","Type":"ContainerDied","Data":"4e3803851bcd5ac03e968cf549d25831bb9acb6416df8cd1945806e16cc49a72"}
Nov 03 20:12:14.873650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.873550    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2" podStartSLOduration=2.873517132 podStartE2EDuration="2.873517132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:14.870832677 +0000 UTC m=+1927.540034148" watchObservedRunningTime="2026-11-03 20:12:14.873517132 +0000 UTC m=+1927.542718605"
Nov 03 20:12:14.876266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.876245    5282 scope.go:122] "RemoveContainer" containerID="13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05"
Nov 03 20:12:14.876611 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:14.876584    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05\": container with ID starting with 13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05 not found: ID does not exist" containerID="13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05"
Nov 03 20:12:14.876702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.876621    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05"} err="failed to get container status \"13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05\": rpc error: code = NotFound desc = could not find container \"13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05\": container with ID starting with 13cfab55fb2758a18fed7a7ef74d89619e80ec3bec3b35128f6c32139fd2dc05 not found: ID does not exist"
Nov 03 20:12:14.876702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.876646    5282 scope.go:122] "RemoveContainer" containerID="693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b"
Nov 03 20:12:14.891634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.891609    5282 scope.go:122] "RemoveContainer" containerID="693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b"
Nov 03 20:12:14.892068 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:14.891998    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b\": container with ID starting with 693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b not found: ID does not exist" containerID="693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b"
Nov 03 20:12:14.892068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.892035    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b"} err="failed to get container status \"693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b\": rpc error: code = NotFound desc = could not find container \"693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b\": container with ID starting with 693d9e489395ab412d7ec4cdaba25a766639a1da42813d5c9a6ddb0f6320761b not found: ID does not exist"
Nov 03 20:12:14.900709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.900156    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"]
Nov 03 20:12:14.900709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.900533    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:14.905051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.905027    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-76wcd"]
Nov 03 20:12:14.922259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.920811    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"]
Nov 03 20:12:14.922259 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:14.921144    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7182a5_3f1e_4ab6_a0fc_aeb5c0b292bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f4e2b4_d910_4268_a01a_ca6e785a6319.slice/crio-4e3803851bcd5ac03e968cf549d25831bb9acb6416df8cd1945806e16cc49a72\": RecentStats: unable to find data in memory cache]"
Nov 03 20:12:14.958132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:14.929963    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-fqkj8"]
Nov 03 20:12:15.154726 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:15.154700    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ae63e0_de07_4d88_a8b7_5021ee8678e7.slice/crio-a6fd3c1451e0e9455a801ab110c5d6960eef01457a541f8d4e3653e54065f0d5 WatchSource:0}: Error finding container a6fd3c1451e0e9455a801ab110c5d6960eef01457a541f8d4e3653e54065f0d5: Status 404 returned error can't find the container with id a6fd3c1451e0e9455a801ab110c5d6960eef01457a541f8d4e3653e54065f0d5
Nov 03 20:12:15.155397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.155376    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"]
Nov 03 20:12:15.227623 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:15.227592    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf777c24d_c5bf_4e9e_a65f_eef2aa170470.slice/crio-0474886e5cfcefe45029b7207ab28b578cd289cd7c3f21857554ad8c4c9c4857 WatchSource:0}: Error finding container 0474886e5cfcefe45029b7207ab28b578cd289cd7c3f21857554ad8c4c9c4857: Status 404 returned error can't find the container with id 0474886e5cfcefe45029b7207ab28b578cd289cd7c3f21857554ad8c4c9c4857
Nov 03 20:12:15.228556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.228534    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"]
Nov 03 20:12:15.230924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.230883    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:15.231000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.230969    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:15.231042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.231012    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:15.274726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.274693    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:15.468079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.468048    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc" path="/var/lib/kubelet/pods/0e7182a5-3f1e-4ab6-a0fc-aeb5c0b292bc/volumes"
Nov 03 20:12:15.469358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.469333    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f4e2b4-d910-4268-a01a-ca6e785a6319" path="/var/lib/kubelet/pods/77f4e2b4-d910-4268-a01a-ca6e785a6319/volumes"
Nov 03 20:12:15.874938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.874901    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" event={"ID":"d5ae63e0-de07-4d88-a8b7-5021ee8678e7","Type":"ContainerStarted","Data":"5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e"}
Nov 03 20:12:15.875121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.874997    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:15.875121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.875031    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" event={"ID":"d5ae63e0-de07-4d88-a8b7-5021ee8678e7","Type":"ContainerStarted","Data":"a6fd3c1451e0e9455a801ab110c5d6960eef01457a541f8d4e3653e54065f0d5"}
Nov 03 20:12:15.876624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.876597    5282 generic.go:356] "Generic (PLEG): container finished" podID="18ff6775-eb23-46df-ad3e-e361447f36d3" containerID="4faf6b80264df00fbc8eb6992c4baebfaab3a835ae38f72d281e29bab6b5d6d3" exitCode=0
Nov 03 20:12:15.876723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.876679    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2" event={"ID":"18ff6775-eb23-46df-ad3e-e361447f36d3","Type":"ContainerDied","Data":"4faf6b80264df00fbc8eb6992c4baebfaab3a835ae38f72d281e29bab6b5d6d3"}
Nov 03 20:12:15.878336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.878316    5282 generic.go:356] "Generic (PLEG): container finished" podID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerID="bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6" exitCode=0
Nov 03 20:12:15.878455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.878394    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" event={"ID":"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2","Type":"ContainerDied","Data":"bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6"}
Nov 03 20:12:15.878859 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.878841    5282 scope.go:122] "RemoveContainer" containerID="bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6"
Nov 03 20:12:15.880776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.880746    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:15.880884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.880821    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:15.880884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.880864    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:15.883060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.883038    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" event={"ID":"f777c24d-c5bf-4e9e-a65f-eef2aa170470","Type":"ContainerStarted","Data":"f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e"}
Nov 03 20:12:15.883177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.883079    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" event={"ID":"f777c24d-c5bf-4e9e-a65f-eef2aa170470","Type":"ContainerStarted","Data":"0474886e5cfcefe45029b7207ab28b578cd289cd7c3f21857554ad8c4c9c4857"}
Nov 03 20:12:15.895530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.895444    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" podStartSLOduration=4.502661335 podStartE2EDuration="4.895431163s" totalImagesPullingTime="392.769828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:15.895093261 +0000 UTC m=+1928.564294732" watchObservedRunningTime="2026-11-03 20:12:15.895431163 +0000 UTC m=+1928.564632634"
Nov 03 20:12:15.900673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.900654    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:15.942860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.942809    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podStartSLOduration=4.942789959 podStartE2EDuration="4.942789959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:15.937834176 +0000 UTC m=+1928.607035646" watchObservedRunningTime="2026-11-03 20:12:15.942789959 +0000 UTC m=+1928.611991430"
Nov 03 20:12:15.990349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:15.990311    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:16.518290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:16.518256    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"]
Nov 03 20:12:16.619136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:16.619109    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:16.739268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:16.739232    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:16.995022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:16.994957    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:16.998749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:16.997183    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:12:16.998749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:16.997263    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 20:12:17.052219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.039245    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" event={"ID":"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2","Type":"ContainerStarted","Data":"05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf"}
Nov 03 20:12:17.175088 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:17.174392    5282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-serving-cert: secret "maas-api-e2e-mt-acd98d21-serving-cert" not found
Nov 03 20:12:17.175088 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:17.174492    5282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls podName:d5ae63e0-de07-4d88-a8b7-5021ee8678e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:12:17.674470148 +0000 UTC m=+1930.343671597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls") pod "maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" (UID: "d5ae63e0-de07-4d88-a8b7-5021ee8678e7") : secret "maas-api-e2e-mt-acd98d21-serving-cert" not found
Nov 03 20:12:17.206314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.206273    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"]
Nov 03 20:12:17.307277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.297973    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:17.738280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.715413    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:12:17.738280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.715485    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 20:12:17.738280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.723478    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:17.738280 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:17.726782    5282 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-mt-acd98d21-c297v\" is forbidden: User \"system:node:ip-10-0-142-244.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-244.us-east-2.compute.internal' and this object" podUID="d28ea5ce-1fd2-4252-988b-7754c0442282" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"
Nov 03 20:12:17.782749 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:17.770904    5282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-serving-cert: secret "maas-api-e2e-mt-acd98d21-serving-cert" not found
Nov 03 20:12:17.782749 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:17.770961    5282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls podName:d5ae63e0-de07-4d88-a8b7-5021ee8678e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:12:18.770948077 +0000 UTC m=+1931.440149538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls") pod "maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" (UID: "d5ae63e0-de07-4d88-a8b7-5021ee8678e7") : secret "maas-api-e2e-mt-acd98d21-serving-cert" not found
Nov 03 20:12:17.782749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.772029    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:12:17.782749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.773036    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-acd98d21-c297v"]
Nov 03 20:12:17.782749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.773078    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:12:17.921222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.921025    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:12:17.921222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:17.921084    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 20:12:18.194545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.194504    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:18.201888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.201856    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:18.219513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.219476    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2" event={"ID":"18ff6775-eb23-46df-ad3e-e361447f36d3","Type":"ContainerDied","Data":"2a986692aad21440708eb921ee943f972c3db724763fc6e6c075969bfa8e72be"}
Nov 03 20:12:18.219764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.219747    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a986692aad21440708eb921ee943f972c3db724763fc6e6c075969bfa8e72be"
Nov 03 20:12:18.220068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.220038    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" podUID="d5ae63e0-de07-4d88-a8b7-5021ee8678e7" containerName="maas-api" containerID="cri-o://5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e" gracePeriod=30
Nov 03 20:12:18.319221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.304542    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-xks-ca\") pod \"18ff6775-eb23-46df-ad3e-e361447f36d3\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") "
Nov 03 20:12:18.319221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.304604    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-openshift-ca\") pod \"18ff6775-eb23-46df-ad3e-e361447f36d3\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") "
Nov 03 20:12:18.319221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.304637    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ff6775-eb23-46df-ad3e-e361447f36d3-kube-api-access-glhnl\" (UniqueName: \"kubernetes.io/projected/18ff6775-eb23-46df-ad3e-e361447f36d3-kube-api-access-glhnl\") pod \"18ff6775-eb23-46df-ad3e-e361447f36d3\" (UID: \"18ff6775-eb23-46df-ad3e-e361447f36d3\") "
Nov 03 20:12:18.335078 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.333029    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-openshift-ca" pod "18ff6775-eb23-46df-ad3e-e361447f36d3" (UID: "18ff6775-eb23-46df-ad3e-e361447f36d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:18.354362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.352438    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ff6775-eb23-46df-ad3e-e361447f36d3-kube-api-access-glhnl" pod "18ff6775-eb23-46df-ad3e-e361447f36d3" (UID: "18ff6775-eb23-46df-ad3e-e361447f36d3"). InnerVolumeSpecName "kube-api-access-glhnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:18.354362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.352516    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-xks-ca" pod "18ff6775-eb23-46df-ad3e-e361447f36d3" (UID: "18ff6775-eb23-46df-ad3e-e361447f36d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:18.431241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.428063    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:18.431241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.428101    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ff6775-eb23-46df-ad3e-e361447f36d3-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:18.431241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.428116    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhnl\" (UniqueName: \"kubernetes.io/projected/18ff6775-eb23-46df-ad3e-e361447f36d3-kube-api-access-glhnl\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:18.605912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.594890    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:12:18.605912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.594954    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 20:12:18.660653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.660613    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:18.737963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.737934    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:18.843047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.842953    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"]
Nov 03 20:12:18.844978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.844946    5282 cpu_manager.go:414] "containerMap: removing container" podUID="d5ae63e0-de07-4d88-a8b7-5021ee8678e7" containerName="maas-api"
Nov 03 20:12:18.845095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.844978    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ae63e0-de07-4d88-a8b7-5021ee8678e7" containerName="maas-api"
Nov 03 20:12:18.845161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.845100    5282 cpu_manager.go:414] "containerMap: removing container" podUID="18ff6775-eb23-46df-ad3e-e361447f36d3" containerName="revoke-keys"
Nov 03 20:12:18.845161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.845109    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ff6775-eb23-46df-ad3e-e361447f36d3" containerName="revoke-keys"
Nov 03 20:12:18.845789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.845762    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ff6775-eb23-46df-ad3e-e361447f36d3" containerName="revoke-keys"
Nov 03 20:12:18.845890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.845817    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ae63e0-de07-4d88-a8b7-5021ee8678e7" containerName="maas-api"
Nov 03 20:12:18.851134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.851109    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls\") pod \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") "
Nov 03 20:12:18.851829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.851803    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"]
Nov 03 20:12:18.852063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.852021    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:18.853051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.852113    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-metrics-certs\") pod \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") "
Nov 03 20:12:18.853051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.852573    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-openshift-service-ca\") pod \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") "
Nov 03 20:12:18.853051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.852633    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-kube-api-access-w9nhb\" (UniqueName: \"kubernetes.io/projected/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-kube-api-access-w9nhb\") pod \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\" (UID: \"d5ae63e0-de07-4d88-a8b7-5021ee8678e7\") "
Nov 03 20:12:18.853220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.853156    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-openshift-service-ca" pod "d5ae63e0-de07-4d88-a8b7-5021ee8678e7" (UID: "d5ae63e0-de07-4d88-a8b7-5021ee8678e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:18.853812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.853778    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:18.855043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.854919    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-metrics-certs" pod "d5ae63e0-de07-4d88-a8b7-5021ee8678e7" (UID: "d5ae63e0-de07-4d88-a8b7-5021ee8678e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:18.857615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.856315    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls" pod "d5ae63e0-de07-4d88-a8b7-5021ee8678e7" (UID: "d5ae63e0-de07-4d88-a8b7-5021ee8678e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:18.857615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.856432    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-kube-api-access-w9nhb" pod "d5ae63e0-de07-4d88-a8b7-5021ee8678e7" (UID: "d5ae63e0-de07-4d88-a8b7-5021ee8678e7"). InnerVolumeSpecName "kube-api-access-w9nhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:18.857615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.856669    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-facc79-serving-cert\""
Nov 03 20:12:18.967291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.911120    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:12:18.967291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:18.911182    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 20:12:19.062298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.062251    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.062789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.062764    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5msm\" (UniqueName: \"kubernetes.io/projected/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-kube-api-access-g5msm\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.064367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.064339    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-metrics-certs\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.064577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.064559    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-maas-api-tls\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.070219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.064957    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:19.070430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.070408    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:19.070544 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.070531    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nhb\" (UniqueName: \"kubernetes.io/projected/d5ae63e0-de07-4d88-a8b7-5021ee8678e7-kube-api-access-w9nhb\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:19.095847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.095755    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:19.188257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.174255    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.188257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.174437    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5msm\" (UniqueName: \"kubernetes.io/projected/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-kube-api-access-g5msm\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.188257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.174568    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-metrics-certs\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.188257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.174621    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-maas-api-tls\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.214220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.191834    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-openshift-service-ca\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.241266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.239813    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-maas-api-tls\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.265217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.262981    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5msm\" (UniqueName: \"kubernetes.io/projected/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-kube-api-access-g5msm\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.284094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.273664    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:19.343872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.343818    5282 generic.go:356] "Generic (PLEG): container finished" podID="d5ae63e0-de07-4d88-a8b7-5021ee8678e7" containerID="5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e" exitCode=0
Nov 03 20:12:19.351403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.351311    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"
Nov 03 20:12:19.360378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.360335    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"
Nov 03 20:12:19.366238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.363773    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-metrics-certs\") pod \"maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.366238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.363843    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" event={"ID":"d5ae63e0-de07-4d88-a8b7-5021ee8678e7","Type":"ContainerDied","Data":"5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e"}
Nov 03 20:12:19.366238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.363900    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w" event={"ID":"d5ae63e0-de07-4d88-a8b7-5021ee8678e7","Type":"ContainerDied","Data":"a6fd3c1451e0e9455a801ab110c5d6960eef01457a541f8d4e3653e54065f0d5"}
Nov 03 20:12:19.366238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.363930    5282 scope.go:122] "RemoveContainer" containerID="5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e"
Nov 03 20:12:19.432085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.432014    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:19.483293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.483250    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:19.534256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.532863    5282 scope.go:122] "RemoveContainer" containerID="5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e"
Nov 03 20:12:19.598217 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:19.578106    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e\": container with ID starting with 5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e not found: ID does not exist" containerID="5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e"
Nov 03 20:12:19.598217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.578150    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e"} err="failed to get container status \"5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e\": rpc error: code = NotFound desc = could not find container \"5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e\": container with ID starting with 5b95cef7df0d914c97328c557ef7504d42121c088fbebe32ea9d5a9f77b76a7e not found: ID does not exist"
Nov 03 20:12:19.598217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.593934    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:12:19.598217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.593990    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 20:12:19.631319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.621096    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28ea5ce-1fd2-4252-988b-7754c0442282" path="/var/lib/kubelet/pods/d28ea5ce-1fd2-4252-988b-7754c0442282/volumes"
Nov 03 20:12:19.728451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.689459    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:19.736454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.729024    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"]
Nov 03 20:12:19.736454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.731281    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cd7f5c-dj9rf"]
Nov 03 20:12:19.736454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.731317    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"]
Nov 03 20:12:19.736454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.731349    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-acd98d21-7d6f98b89b-72d4w"]
Nov 03 20:12:19.869115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.869073    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:19.921274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.913357    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:12:19.921274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:19.913427    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 20:12:20.091222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.090589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:20.393248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.388401    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:12:20.591220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.590880    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:12:20.591220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.590935    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 20:12:20.813437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.804240    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"]
Nov 03 20:12:20.909283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.908708    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:20.911042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.911015    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:12:20.911276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.911249    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 20:12:20.928674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:20.928643    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:12:21.221276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.219703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:21.460151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.460122    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d356215-1ebc-4bd1-85ed-16245f324280" path="/var/lib/kubelet/pods/9d356215-1ebc-4bd1-85ed-16245f324280/volumes"
Nov 03 20:12:21.460637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.460620    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ae63e0-de07-4d88-a8b7-5021ee8678e7" path="/var/lib/kubelet/pods/d5ae63e0-de07-4d88-a8b7-5021ee8678e7/volumes"
Nov 03 20:12:21.595169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.594713    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:12:21.595169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.594770    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 20:12:21.667832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.667798    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c" event={"ID":"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11","Type":"ContainerStarted","Data":"51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f"}
Nov 03 20:12:21.668026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.667846    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c" event={"ID":"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11","Type":"ContainerStarted","Data":"f3f03f08825f2ee7f73eee4fdccff025690d0416d5b37d88065e9d5a690b47a2"}
Nov 03 20:12:21.668026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.667940    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:21.702221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.699437    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c" podStartSLOduration=10.08941753 podStartE2EDuration="10.699417135s" totalImagesPullingTime="609.999605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:21.697632768 +0000 UTC m=+1934.366834240" watchObservedRunningTime="2026-11-03 20:12:21.699417135 +0000 UTC m=+1934.368618606"
Nov 03 20:12:21.722051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.721987    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:21.909284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.901603    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:12:21.909284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:21.901663    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 20:12:22.328289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.324137    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:22.353223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.349263    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:12:22.440222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.439227    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:22.591276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.590247    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:12:22.591276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.590308    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 20:12:22.635219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.635006    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:22.922277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.912968    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 20:12:22.922277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:22.913043    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 20:12:23.008225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:23.008064    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:23.606377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:23.605591    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 03 20:12:23.606377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:23.605658    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 03 20:12:23.800829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:23.800791    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:23.945276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:23.941679    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:23.945276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:23.943530    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:23.958223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:23.955213    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:12:24.475635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:24.475599    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:24.641741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:24.641519    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:12:24.649734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:24.648421    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:12:25.212473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:25.212439    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:25.784647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:25.784616    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:12:26.053587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:26.053464    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:26.240218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:26.231776    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:26.373958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:26.373866    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"]
Nov 03 20:12:26.374827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:26.374786    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy" containerID="cri-o://8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0" gracePeriod=30
Nov 03 20:12:26.480720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:26.472588    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:26.779923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:26.773932    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:27.402223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:27.396762    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:27.702397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:27.702326    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:12:27.716479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:27.716448    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:28.162193 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:28.152289    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:28.619256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:28.617092    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:29.608539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:29.608506    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:30.061980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:30.061948    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:30.682443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:30.681941    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:31.086219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.085828    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:31.450076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.449957    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:31.620253 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:31.620209    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:12:31.785898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.785868    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:12:31.833970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.833931    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"]
Nov 03 20:12:31.836729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.836697    5282 cpu_manager.go:414] "containerMap: removing container" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy"
Nov 03 20:12:31.836930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.836912    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy"
Nov 03 20:12:31.837081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.837068    5282 cpu_manager.go:414] "containerMap: removing container" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy"
Nov 03 20:12:31.837171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.837159    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy"
Nov 03 20:12:31.837705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.837683    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy"
Nov 03 20:12:31.837853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.837839    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerName="istio-proxy"
Nov 03 20:12:31.848631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.848593    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:31.856896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.856871    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f817b0-openshift-default-dockercfg-wzqz5\""
Nov 03 20:12:31.873165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.873132    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"]
Nov 03 20:12:31.876459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.876423    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:31.902499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.902456    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-envoy\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.902759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.902730    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-data\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.902920 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.902783    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-podinfo\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.902920 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.902895    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-credential-socket\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.903083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.902986    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istiod-ca-cert\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.903232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.903083    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-socket\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.903330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.903310    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-token\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-token\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.903474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.903455    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-kube-api-access-4g9dt\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-kube-api-access-4g9dt\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.903682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.903661    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-certs\") pod \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\" (UID: \"71bb8111-a2fd-4e84-a7da-e63466d8b6d9\") "
Nov 03 20:12:31.903761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.903718    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-data" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:31.904295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.904261    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-socket" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:31.904584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.904560    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istiod-ca-cert" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:31.904685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.904660    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:31.904756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.904693    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:31.904826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.904803    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-credential-socket" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:31.904881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.904864    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-certs" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:31.906387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.906353    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-token" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:31.906503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.906441    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-envoy" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:31.906625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.906601    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-kube-api-access-4g9dt" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "kube-api-access-4g9dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:31.907108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:31.907049    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-podinfo" pod "71bb8111-a2fd-4e84-a7da-e63466d8b6d9" (UID: "71bb8111-a2fd-4e84-a7da-e63466d8b6d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.009979    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-envoy\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010055    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010094    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxn7\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-kube-api-access-wcxn7\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010268    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-credential-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010340    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-token\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010389    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-data\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010473    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-certs\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010568    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-podinfo\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010593    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60336029-d74c-4227-a77f-7f2bfc2a17f1-istiod-ca-cert\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010738    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010756    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010770    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g9dt\" (UniqueName: \"kubernetes.io/projected/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-kube-api-access-4g9dt\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010785    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010799    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010813    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.012300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.010826    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71bb8111-a2fd-4e84-a7da-e63466d8b6d9-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.038235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.037737    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.115684    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-envoy\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.115765    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.115805    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxn7\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-kube-api-access-wcxn7\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.115973    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-credential-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.116046    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-token\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.116098    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-data\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.116188    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-certs\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.116341    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-podinfo\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.117221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.116368    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60336029-d74c-4227-a77f-7f2bfc2a17f1-istiod-ca-cert\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.118125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.117231    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60336029-d74c-4227-a77f-7f2bfc2a17f1-istiod-ca-cert\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.118293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.118221    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.136236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.118734    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-credential-socket\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.136236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.118974    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-certs\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.136236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.119221    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-data\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.153150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.153083    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-envoy\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.154333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.154304    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-podinfo\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.156389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.156366    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxn7\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-kube-api-access-wcxn7\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.194375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.194345    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-token\") pod \"e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") " pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.379230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.379126    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:32.505219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.496988    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:32.539222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.533686    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:32.574220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.559393    5282 generic.go:356] "Generic (PLEG): container finished" podID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" containerID="8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0" exitCode=0
Nov 03 20:12:32.574220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.559770    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" event={"ID":"71bb8111-a2fd-4e84-a7da-e63466d8b6d9","Type":"ContainerDied","Data":"8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0"}
Nov 03 20:12:32.574220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.559836    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j" event={"ID":"71bb8111-a2fd-4e84-a7da-e63466d8b6d9","Type":"ContainerDied","Data":"aa58c9449a1bcd981e2a253f3b70a57f497809f1fff85354fea7a915a6aaf0f5"}
Nov 03 20:12:32.574220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.559866    5282 scope.go:122] "RemoveContainer" containerID="8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0"
Nov 03 20:12:32.574220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.560059    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"
Nov 03 20:12:32.721677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.721643    5282 scope.go:122] "RemoveContainer" containerID="124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4"
Nov 03 20:12:32.824440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.820247    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"]
Nov 03 20:12:32.826548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.826481    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cd7f5c-openshift-default-f7d549ddc-lfp5j"]
Nov 03 20:12:32.963161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.963136    5282 scope.go:122] "RemoveContainer" containerID="8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0"
Nov 03 20:12:32.969680 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:32.969650    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0\": container with ID starting with 8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0 not found: ID does not exist" containerID="8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0"
Nov 03 20:12:32.969872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.969846    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0"} err="failed to get container status \"8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0\": rpc error: code = NotFound desc = could not find container \"8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0\": container with ID starting with 8e686fccc1599cfdb560593f7b525551c395b203ac963132ffb6e4f96ccb79f0 not found: ID does not exist"
Nov 03 20:12:32.969981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.969968    5282 scope.go:122] "RemoveContainer" containerID="124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4"
Nov 03 20:12:32.975407 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:32.975367    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4\": container with ID starting with 124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4 not found: ID does not exist" containerID="124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4"
Nov 03 20:12:32.975549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:32.975414    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4"} err="failed to get container status \"124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4\": rpc error: code = NotFound desc = could not find container \"124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4\": container with ID starting with 124f3c7d9c736b832cdd58181523b23b48e59c54cefd1940a8d105c8bea036d4 not found: ID does not exist"
Nov 03 20:12:33.296397 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:33.296366    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60336029_d74c_4227_a77f_7f2bfc2a17f1.slice/crio-3e78e078c58660aa65f14d083a1902f0b0c52f42043eeecdbafe6521e21bc299 WatchSource:0}: Error finding container 3e78e078c58660aa65f14d083a1902f0b0c52f42043eeecdbafe6521e21bc299: Status 404 returned error can't find the container with id 3e78e078c58660aa65f14d083a1902f0b0c52f42043eeecdbafe6521e21bc299
Nov 03 20:12:33.297699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.297671    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"]
Nov 03 20:12:33.299246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.299209    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:33.299316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.299279    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:33.299350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.299317    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:33.460703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.460667    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71bb8111-a2fd-4e84-a7da-e63466d8b6d9" path="/var/lib/kubelet/pods/71bb8111-a2fd-4e84-a7da-e63466d8b6d9/volumes"
Nov 03 20:12:33.591136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.591056    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" event={"ID":"60336029-d74c-4227-a77f-7f2bfc2a17f1","Type":"ContainerStarted","Data":"2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172"}
Nov 03 20:12:33.591136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.591106    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" event={"ID":"60336029-d74c-4227-a77f-7f2bfc2a17f1","Type":"ContainerStarted","Data":"3e78e078c58660aa65f14d083a1902f0b0c52f42043eeecdbafe6521e21bc299"}
Nov 03 20:12:33.628406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.628346    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podStartSLOduration=2.6283270549999997 podStartE2EDuration="2.628327055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:33.622918147 +0000 UTC m=+1946.292119617" watchObservedRunningTime="2026-11-03 20:12:33.628327055 +0000 UTC m=+1946.297528527"
Nov 03 20:12:33.656081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.656051    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:33.700937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.700899    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"]
Nov 03 20:12:33.707653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.707627    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:33.713758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.713714    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-663013-dockercfg-mxskt\""
Nov 03 20:12:33.713990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.713761    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-663013\""
Nov 03 20:12:33.717839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.717814    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"]
Nov 03 20:12:33.776625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.776582    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:33.789266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.789234    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e2c8d-47b1-4787-9efb-393780359cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:33.789539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.789515    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsvj\" (UniqueName: \"kubernetes.io/projected/030e2c8d-47b1-4787-9efb-393780359cc1-kube-api-access-5rsvj\") pod \"payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:33.897267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.896014    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e2c8d-47b1-4787-9efb-393780359cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:33.897267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.896282    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsvj\" (UniqueName: \"kubernetes.io/projected/030e2c8d-47b1-4787-9efb-393780359cc1-kube-api-access-5rsvj\") pod \"payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:33.909220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.897717    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e2c8d-47b1-4787-9efb-393780359cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:33.991221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:33.986976    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsvj\" (UniqueName: \"kubernetes.io/projected/030e2c8d-47b1-4787-9efb-393780359cc1-kube-api-access-5rsvj\") pod \"payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:34.019222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:34.014229    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:34.061352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:34.059732    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:34.502291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:34.502256    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:34.508497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:34.508470    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:34.599866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:34.599787    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:34.600663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:34.600639    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:12:34.637905 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:34.637878    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030e2c8d_47b1_4787_9efb_393780359cc1.slice/crio-ff5938afab72419321961539ca72157a40354f58f7afc9768310b38155bd7c4e WatchSource:0}: Error finding container ff5938afab72419321961539ca72157a40354f58f7afc9768310b38155bd7c4e: Status 404 returned error can't find the container with id ff5938afab72419321961539ca72157a40354f58f7afc9768310b38155bd7c4e
Nov 03 20:12:34.656074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:34.652521    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"]
Nov 03 20:12:35.231308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:35.231268    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:35.609230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:35.609120    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" event={"ID":"030e2c8d-47b1-4787-9efb-393780359cc1","Type":"ContainerStarted","Data":"34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c"}
Nov 03 20:12:35.609230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:35.609173    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" event={"ID":"030e2c8d-47b1-4787-9efb-393780359cc1","Type":"ContainerStarted","Data":"ff5938afab72419321961539ca72157a40354f58f7afc9768310b38155bd7c4e"}
Nov 03 20:12:35.609704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:35.609358    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:35.628164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:35.628119    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" podStartSLOduration=2.628106599 podStartE2EDuration="2.628106599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:35.625654769 +0000 UTC m=+1948.294856252" watchObservedRunningTime="2026-11-03 20:12:35.628106599 +0000 UTC m=+1948.297308111"
Nov 03 20:12:36.006619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:36.006573    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:36.146478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:36.146444    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:36.296293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:36.296101    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:37.247741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:37.247703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:37.517268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:37.514122    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:37.913274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:37.908797    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:38.076220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.075255    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:38.327272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.301559    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"]
Nov 03 20:12:38.358584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.358552    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.434226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.427670    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbs4g\" (UniqueName: \"kubernetes.io/projected/477c28ea-7bed-4489-9bbc-5c0b812b317e-kube-api-access-bbs4g\") pod \"payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.434226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.427939    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477c28ea-7bed-4489-9bbc-5c0b812b317e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.497491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.473595    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f817b0-dockercfg-664v5\""
Nov 03 20:12:38.497491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.473795    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f817b0\""
Nov 03 20:12:38.499800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.499764    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"]
Nov 03 20:12:38.582266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.539720    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbs4g\" (UniqueName: \"kubernetes.io/projected/477c28ea-7bed-4489-9bbc-5c0b812b317e-kube-api-access-bbs4g\") pod \"payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.582266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.540023    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477c28ea-7bed-4489-9bbc-5c0b812b317e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.582266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.551522    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477c28ea-7bed-4489-9bbc-5c0b812b317e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.599895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.599278    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbs4g\" (UniqueName: \"kubernetes.io/projected/477c28ea-7bed-4489-9bbc-5c0b812b317e-kube-api-access-bbs4g\") pod \"payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.625089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.621705    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:38.811602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.811572    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"]
Nov 03 20:12:38.819602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.819570    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.830560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.830534    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"]
Nov 03 20:12:38.830858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.830834    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49fb2b\"/\"test-model-49fb2b-kserve-self-signed-certs\""
Nov 03 20:12:38.830858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.830835    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49fb2b\"/\"default-dockercfg-gdppt\""
Nov 03 20:12:38.830999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.830853    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49fb2b\"/\"kube-root-ca.crt\""
Nov 03 20:12:38.830999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.830899    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49fb2b\"/\"openshift-service-ca.crt\""
Nov 03 20:12:38.847808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.847735    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tmp-dir\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.847950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.847816    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tls-certs\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.848105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.848017    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwgf\" (UniqueName: \"kubernetes.io/projected/b05d5324-4bed-47b0-8dbd-3af0168a51b4-kube-api-access-dqwgf\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.848221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.848118    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-home\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.848380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.848356    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-model-cache\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.848472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.848457    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-dshm\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.923260 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:38.923172    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477c28ea_7bed_4489_9bbc_5c0b812b317e.slice/crio-f27e9f6c3a573bdf1d91263bf0af8721fcf2b3c63ea190f3858bc9925ac088c8 WatchSource:0}: Error finding container f27e9f6c3a573bdf1d91263bf0af8721fcf2b3c63ea190f3858bc9925ac088c8: Status 404 returned error can't find the container with id f27e9f6c3a573bdf1d91263bf0af8721fcf2b3c63ea190f3858bc9925ac088c8
Nov 03 20:12:38.923805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.923769    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"]
Nov 03 20:12:38.924014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.923953    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:38.950478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.950453    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwgf\" (UniqueName: \"kubernetes.io/projected/b05d5324-4bed-47b0-8dbd-3af0168a51b4-kube-api-access-dqwgf\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.950594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.950500    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-home\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.950651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.950613    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-model-cache\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.950704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.950672    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-dshm\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.950854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.950831    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tmp-dir\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.950993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.950880    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tls-certs\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.950993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.950979    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-home\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.951111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.951087    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-model-cache\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.951322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.951301    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tmp-dir\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.953019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.953001    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-dshm\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.953448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.953431    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tls-certs\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:38.958434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:38.958412    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwgf\" (UniqueName: \"kubernetes.io/projected/b05d5324-4bed-47b0-8dbd-3af0168a51b4-kube-api-access-dqwgf\") pod \"test-model-49fb2b-kserve-6b4958b7fc-52tt7\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") " pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:39.066229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:39.066042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:39.139972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:39.139629    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:39.251005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:39.250913    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" event={"ID":"477c28ea-7bed-4489-9bbc-5c0b812b317e","Type":"ContainerStarted","Data":"f27e9f6c3a573bdf1d91263bf0af8721fcf2b3c63ea190f3858bc9925ac088c8"}
Nov 03 20:12:39.620296 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:39.620254    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb05d5324_4bed_47b0_8dbd_3af0168a51b4.slice/crio-f3d890310a95265a0236c9b81e0da440f4ae0e3eb2eed27f9ffbcb93452f3012 WatchSource:0}: Error finding container f3d890310a95265a0236c9b81e0da440f4ae0e3eb2eed27f9ffbcb93452f3012: Status 404 returned error can't find the container with id f3d890310a95265a0236c9b81e0da440f4ae0e3eb2eed27f9ffbcb93452f3012
Nov 03 20:12:39.620807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:39.620783    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"]
Nov 03 20:12:39.793555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:39.793524    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:40.260710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:40.260671    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" event={"ID":"477c28ea-7bed-4489-9bbc-5c0b812b317e","Type":"ContainerStarted","Data":"c9102ac891b92ed4835ff2634bf3b6d7e6019a9c3977be6105070464ddd50d6e"}
Nov 03 20:12:40.260887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:40.260745    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:40.262374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:40.262350    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" event={"ID":"b05d5324-4bed-47b0-8dbd-3af0168a51b4","Type":"ContainerStarted","Data":"535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89"}
Nov 03 20:12:40.262505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:40.262379    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" event={"ID":"b05d5324-4bed-47b0-8dbd-3af0168a51b4","Type":"ContainerStarted","Data":"f3d890310a95265a0236c9b81e0da440f4ae0e3eb2eed27f9ffbcb93452f3012"}
Nov 03 20:12:40.284298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:40.284246    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" podStartSLOduration=3.284234935 podStartE2EDuration="3.284234935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:40.279972094 +0000 UTC m=+1952.949173563" watchObservedRunningTime="2026-11-03 20:12:40.284234935 +0000 UTC m=+1952.953436405"
Nov 03 20:12:40.299513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:40.299470    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podStartSLOduration=2.299458963 podStartE2EDuration="2.299458963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:40.295456718 +0000 UTC m=+1952.964658212" watchObservedRunningTime="2026-11-03 20:12:40.299458963 +0000 UTC m=+1952.968660463"
Nov 03 20:12:40.575771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:40.571707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:41.140937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:41.140903    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:41.158978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:41.158950    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"]
Nov 03 20:12:41.171410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:41.171376    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"]
Nov 03 20:12:41.267961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:41.267915    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:41.389398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:41.389363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:41.535220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:41.523986    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:42.232147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:42.232107    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:43.046221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:43.039349    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:43.952587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:43.952552    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:44.981221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:44.970303    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:45.824256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:45.824220    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:46.159581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.159494    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8" podUID="13c11ce5-b830-4da0-87fe-0f57fc49aeb2" containerName="payload-processing" containerID="cri-o://5c3aad9ad5d0586d7f9bff5cd7a695b9358642e4215407f12eed2016562be9c0" gracePeriod=25
Nov 03 20:12:46.171930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.171892    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx" podUID="86f98f1f-2e73-466d-9ba7-9725790a772a" containerName="payload-pre-processing" containerID="cri-o://ac52ed2ec8cbff32573d84e063d090547e01266f5abf1fa50f40842d0143229f" gracePeriod=25
Nov 03 20:12:46.432446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.430913    5282 generic.go:356] "Generic (PLEG): container finished" podID="13c11ce5-b830-4da0-87fe-0f57fc49aeb2" containerID="5c3aad9ad5d0586d7f9bff5cd7a695b9358642e4215407f12eed2016562be9c0" exitCode=0
Nov 03 20:12:46.432446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.431097    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8" event={"ID":"13c11ce5-b830-4da0-87fe-0f57fc49aeb2","Type":"ContainerDied","Data":"5c3aad9ad5d0586d7f9bff5cd7a695b9358642e4215407f12eed2016562be9c0"}
Nov 03 20:12:46.433962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.433768    5282 generic.go:356] "Generic (PLEG): container finished" podID="86f98f1f-2e73-466d-9ba7-9725790a772a" containerID="ac52ed2ec8cbff32573d84e063d090547e01266f5abf1fa50f40842d0143229f" exitCode=0
Nov 03 20:12:46.433962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.433875    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx" event={"ID":"86f98f1f-2e73-466d-9ba7-9725790a772a","Type":"ContainerDied","Data":"ac52ed2ec8cbff32573d84e063d090547e01266f5abf1fa50f40842d0143229f"}
Nov 03 20:12:46.554860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.554826    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:12:46.641671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.641642    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:12:46.662076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.662031    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:12:46.710441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.710354    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:46.727324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.727120    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-plugins-config-volume\") pod \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") "
Nov 03 20:12:46.727637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.727572    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-plugins-config-volume" pod "13c11ce5-b830-4da0-87fe-0f57fc49aeb2" (UID: "13c11ce5-b830-4da0-87fe-0f57fc49aeb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:46.727916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.727897    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-kube-api-access-gjjhs\" (UniqueName: \"kubernetes.io/projected/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-kube-api-access-gjjhs\") pod \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\" (UID: \"13c11ce5-b830-4da0-87fe-0f57fc49aeb2\") "
Nov 03 20:12:46.728893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.728860    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:46.731047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.731020    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-kube-api-access-gjjhs" pod "13c11ce5-b830-4da0-87fe-0f57fc49aeb2" (UID: "13c11ce5-b830-4da0-87fe-0f57fc49aeb2"). InnerVolumeSpecName "kube-api-access-gjjhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:46.883231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.879009    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f98f1f-2e73-466d-9ba7-9725790a772a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f98f1f-2e73-466d-9ba7-9725790a772a-plugins-config-volume\") pod \"86f98f1f-2e73-466d-9ba7-9725790a772a\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") "
Nov 03 20:12:46.883231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.879390    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f98f1f-2e73-466d-9ba7-9725790a772a-kube-api-access-48cvv\" (UniqueName: \"kubernetes.io/projected/86f98f1f-2e73-466d-9ba7-9725790a772a-kube-api-access-48cvv\") pod \"86f98f1f-2e73-466d-9ba7-9725790a772a\" (UID: \"86f98f1f-2e73-466d-9ba7-9725790a772a\") "
Nov 03 20:12:46.883231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.880356    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjhs\" (UniqueName: \"kubernetes.io/projected/13c11ce5-b830-4da0-87fe-0f57fc49aeb2-kube-api-access-gjjhs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:46.949250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.944432    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f98f1f-2e73-466d-9ba7-9725790a772a-kube-api-access-48cvv" pod "86f98f1f-2e73-466d-9ba7-9725790a772a" (UID: "86f98f1f-2e73-466d-9ba7-9725790a772a"). InnerVolumeSpecName "kube-api-access-48cvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:46.949250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.946297    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f98f1f-2e73-466d-9ba7-9725790a772a-plugins-config-volume" pod "86f98f1f-2e73-466d-9ba7-9725790a772a" (UID: "86f98f1f-2e73-466d-9ba7-9725790a772a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:46.999220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.989290    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f98f1f-2e73-466d-9ba7-9725790a772a-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:46.999220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:46.989325    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48cvv\" (UniqueName: \"kubernetes.io/projected/86f98f1f-2e73-466d-9ba7-9725790a772a-kube-api-access-48cvv\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:47.336916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.336834    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"]
Nov 03 20:12:47.337400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.337094    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv" podUID="c3a3038a-192a-4c85-997a-daf9a0734b38" containerName="maas-api" containerID="cri-o://3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa" gracePeriod=30
Nov 03 20:12:47.411777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.411748    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:47.445560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.445515    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"
Nov 03 20:12:47.445777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.445747    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx" event={"ID":"86f98f1f-2e73-466d-9ba7-9725790a772a","Type":"ContainerDied","Data":"00645b4e49b7a96c7383bbc91690316936131c1a62b04779ad9f88dfe9010c01"}
Nov 03 20:12:47.445864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.445819    5282 scope.go:122] "RemoveContainer" containerID="ac52ed2ec8cbff32573d84e063d090547e01266f5abf1fa50f40842d0143229f"
Nov 03 20:12:47.466682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.466657    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"
Nov 03 20:12:47.468048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.468018    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8" event={"ID":"13c11ce5-b830-4da0-87fe-0f57fc49aeb2","Type":"ContainerDied","Data":"ff2b451e419e530a22ee6e5440f2629eab4938ef9c19a11a193e1c5b0ee8e4a8"}
Nov 03 20:12:47.469724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.469704    5282 scope.go:122] "RemoveContainer" containerID="5c3aad9ad5d0586d7f9bff5cd7a695b9358642e4215407f12eed2016562be9c0"
Nov 03 20:12:47.495359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.495329    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"]
Nov 03 20:12:47.504047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.504028    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5b918c5d-65c99f94c9-f5fqx"]
Nov 03 20:12:47.514973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.514945    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"]
Nov 03 20:12:47.521343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.521319    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5b918c5d-5fb474f47d-mvtv8"]
Nov 03 20:12:47.956059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:47.955847    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:48.094179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.094150    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:48.101643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.101614    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:12:48.209589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.209510    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a3038a-192a-4c85-997a-daf9a0734b38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a3038a-192a-4c85-997a-daf9a0734b38-openshift-service-ca\") pod \"c3a3038a-192a-4c85-997a-daf9a0734b38\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") "
Nov 03 20:12:48.209589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.209547    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a3038a-192a-4c85-997a-daf9a0734b38-kube-api-access-wh2z7\" (UniqueName: \"kubernetes.io/projected/c3a3038a-192a-4c85-997a-daf9a0734b38-kube-api-access-wh2z7\") pod \"c3a3038a-192a-4c85-997a-daf9a0734b38\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") "
Nov 03 20:12:48.209589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.209583    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-metrics-certs\") pod \"c3a3038a-192a-4c85-997a-daf9a0734b38\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") "
Nov 03 20:12:48.209845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.209826    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-maas-api-tls\") pod \"c3a3038a-192a-4c85-997a-daf9a0734b38\" (UID: \"c3a3038a-192a-4c85-997a-daf9a0734b38\") "
Nov 03 20:12:48.209955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.209926    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a3038a-192a-4c85-997a-daf9a0734b38-openshift-service-ca" pod "c3a3038a-192a-4c85-997a-daf9a0734b38" (UID: "c3a3038a-192a-4c85-997a-daf9a0734b38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:48.211072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.211015    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a3038a-192a-4c85-997a-daf9a0734b38-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:48.211719 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.211694    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-metrics-certs" pod "c3a3038a-192a-4c85-997a-daf9a0734b38" (UID: "c3a3038a-192a-4c85-997a-daf9a0734b38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:48.211816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.211779    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-maas-api-tls" pod "c3a3038a-192a-4c85-997a-daf9a0734b38" (UID: "c3a3038a-192a-4c85-997a-daf9a0734b38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:48.211925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.211906    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a3038a-192a-4c85-997a-daf9a0734b38-kube-api-access-wh2z7" pod "c3a3038a-192a-4c85-997a-daf9a0734b38" (UID: "c3a3038a-192a-4c85-997a-daf9a0734b38"). InnerVolumeSpecName "kube-api-access-wh2z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:48.212965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.212936    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:48.331626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.316691    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh2z7\" (UniqueName: \"kubernetes.io/projected/c3a3038a-192a-4c85-997a-daf9a0734b38-kube-api-access-wh2z7\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:48.355263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.353386    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:48.355884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.355858    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3a3038a-192a-4c85-997a-daf9a0734b38-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:48.470496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.470411    5282 generic.go:356] "Generic (PLEG): container finished" podID="c3a3038a-192a-4c85-997a-daf9a0734b38" containerID="3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa" exitCode=0
Nov 03 20:12:48.470496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.470472    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"
Nov 03 20:12:48.470730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.470523    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv" event={"ID":"c3a3038a-192a-4c85-997a-daf9a0734b38","Type":"ContainerDied","Data":"3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa"}
Nov 03 20:12:48.470730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.470580    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv" event={"ID":"c3a3038a-192a-4c85-997a-daf9a0734b38","Type":"ContainerDied","Data":"1c15cc16988027b856d39386fa36101da04294d4f6c26c53b70f141fb6a0d288"}
Nov 03 20:12:48.470730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.470608    5282 scope.go:122] "RemoveContainer" containerID="3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa"
Nov 03 20:12:48.483649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.483631    5282 scope.go:122] "RemoveContainer" containerID="3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa"
Nov 03 20:12:48.483946 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:48.483923    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa\": container with ID starting with 3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa not found: ID does not exist" containerID="3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa"
Nov 03 20:12:48.484003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.483960    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa"} err="failed to get container status \"3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa\": rpc error: code = NotFound desc = could not find container \"3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa\": container with ID starting with 3afdc1100cb2917a28c7c093458b725c0553d35357c913ca094c5ea5077ea1aa not found: ID does not exist"
Nov 03 20:12:48.501334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.501309    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"]
Nov 03 20:12:48.505676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.505641    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b918c5d-66d488fdc9-589tv"]
Nov 03 20:12:48.513918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:48.513893    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:49.140584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:49.140551    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:49.152412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:49.152386    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:49.327222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:49.323364    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:49.496028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:49.495759    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c11ce5-b830-4da0-87fe-0f57fc49aeb2" path="/var/lib/kubelet/pods/13c11ce5-b830-4da0-87fe-0f57fc49aeb2/volumes"
Nov 03 20:12:49.496586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:49.496453    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f98f1f-2e73-466d-9ba7-9725790a772a" path="/var/lib/kubelet/pods/86f98f1f-2e73-466d-9ba7-9725790a772a/volumes"
Nov 03 20:12:49.497090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:49.497068    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a3038a-192a-4c85-997a-daf9a0734b38" path="/var/lib/kubelet/pods/c3a3038a-192a-4c85-997a-daf9a0734b38/volumes"
Nov 03 20:12:49.508523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:49.508500    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:12:50.109785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:50.109742    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:50.916459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:50.916427    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:51.114156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.114059    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:51.116735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.116672    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"]
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.156732    5282 cpu_manager.go:414] "containerMap: removing container" podUID="13c11ce5-b830-4da0-87fe-0f57fc49aeb2" containerName="payload-processing"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.156762    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c11ce5-b830-4da0-87fe-0f57fc49aeb2" containerName="payload-processing"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.156795    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c3a3038a-192a-4c85-997a-daf9a0734b38" containerName="maas-api"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.156803    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a3038a-192a-4c85-997a-daf9a0734b38" containerName="maas-api"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.156865    5282 cpu_manager.go:414] "containerMap: removing container" podUID="86f98f1f-2e73-466d-9ba7-9725790a772a" containerName="payload-pre-processing"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.156873    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f98f1f-2e73-466d-9ba7-9725790a772a" containerName="payload-pre-processing"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.157123    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f98f1f-2e73-466d-9ba7-9725790a772a" containerName="payload-pre-processing"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.157147    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c11ce5-b830-4da0-87fe-0f57fc49aeb2" containerName="payload-processing"
Nov 03 20:12:51.161172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.157162    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a3038a-192a-4c85-997a-daf9a0734b38" containerName="maas-api"
Nov 03 20:12:51.165278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.165250    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"]
Nov 03 20:12:51.165415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.165361    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.173884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.173783    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-facc79\"/\"kube-root-ca.crt\""
Nov 03 20:12:51.174289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.174138    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-facc79\"/\"default-dockercfg-jv6kl\""
Nov 03 20:12:51.174289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.174176    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-facc79\"/\"openshift-service-ca.crt\""
Nov 03 20:12:51.174289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.174270    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-facc79\"/\"test-model-facc79-kserve-self-signed-certs\""
Nov 03 20:12:51.269160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.263158    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-model-cache\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.269160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.263396    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-home\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.269160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.263550    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb501b02-fb73-487a-ad94-08222d91812c-tls-certs\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.269160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.263581    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhs5\" (UniqueName: \"kubernetes.io/projected/fb501b02-fb73-487a-ad94-08222d91812c-kube-api-access-lmhs5\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.269160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.263710    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-tmp-dir\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.269160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.263737    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-dshm\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.270802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.270668    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:51.333902 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.332438    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:12:51.469953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.469861    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb501b02-fb73-487a-ad94-08222d91812c-tls-certs\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.509882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.502373    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhs5\" (UniqueName: \"kubernetes.io/projected/fb501b02-fb73-487a-ad94-08222d91812c-kube-api-access-lmhs5\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.509882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.502554    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-tmp-dir\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.509882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.502597    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-dshm\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.509882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.502769    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-model-cache\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.509882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.503026    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-home\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.512548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.512519    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-home\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.713521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.609733    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhs5\" (UniqueName: \"kubernetes.io/projected/fb501b02-fb73-487a-ad94-08222d91812c-kube-api-access-lmhs5\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.713521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.652669    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb501b02-fb73-487a-ad94-08222d91812c-tls-certs\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.713521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.655801    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-model-cache\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.730559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.730528    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-dshm\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.785229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.779710    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-tmp-dir\") pod \"test-model-facc79-kserve-ffcf49c85-tf5jv\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") " pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.856131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.789409    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:51.858743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:51.858710    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:52.320380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:52.316248    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:52.702336 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:12:52.702247    5282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.21s"
Nov 03 20:12:52.772029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:52.758458    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:12:52.964107 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:52.964074    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb501b02_fb73_487a_ad94_08222d91812c.slice/crio-3ba77f1fb9fe16636d1564a1b3ec8da6d27f8f303ca836cba08f7a6003c41e7e WatchSource:0}: Error finding container 3ba77f1fb9fe16636d1564a1b3ec8da6d27f8f303ca836cba08f7a6003c41e7e: Status 404 returned error can't find the container with id 3ba77f1fb9fe16636d1564a1b3ec8da6d27f8f303ca836cba08f7a6003c41e7e
Nov 03 20:12:52.964392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:52.964370    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"]
Nov 03 20:12:52.974572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:52.974544    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:53.859359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:53.859312    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:53.912492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:53.912445    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" event={"ID":"fb501b02-fb73-487a-ad94-08222d91812c","Type":"ContainerStarted","Data":"1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79"}
Nov 03 20:12:53.912658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:53.912510    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" event={"ID":"fb501b02-fb73-487a-ad94-08222d91812c","Type":"ContainerStarted","Data":"3ba77f1fb9fe16636d1564a1b3ec8da6d27f8f303ca836cba08f7a6003c41e7e"}
Nov 03 20:12:53.940891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:53.940841    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podStartSLOduration=2.940826082 podStartE2EDuration="2.940826082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:53.935802148 +0000 UTC m=+1966.605003617" watchObservedRunningTime="2026-11-03 20:12:53.940826082 +0000 UTC m=+1966.610027551"
Nov 03 20:12:53.983187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:53.983154    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:54.800738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:54.800708    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:12:54.922650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:54.922618    5282 generic.go:356] "Generic (PLEG): container finished" podID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerID="523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a" exitCode=0
Nov 03 20:12:54.923153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:54.922691    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" event={"ID":"3c004d68-fd7d-4827-ab19-fed3800b9e07","Type":"ContainerDied","Data":"523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a"}
Nov 03 20:12:54.923415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:54.923395    5282 scope.go:122] "RemoveContainer" containerID="523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a"
Nov 03 20:12:54.925400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:54.925366    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:54.925522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:54.925458    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:54.925522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:54.925504    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:55.006301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:55.006277    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:55.174868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:55.174834    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:55.312345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:55.312154    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"]
Nov 03 20:12:55.437960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:55.437244    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:55.990261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:55.990218    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerName="istio-proxy" containerID="cri-o://6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9" gracePeriod=30
Nov 03 20:12:55.991251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:55.991168    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" event={"ID":"3c004d68-fd7d-4827-ab19-fed3800b9e07","Type":"ContainerStarted","Data":"6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9"}
Nov 03 20:12:56.203222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.202832    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:56.449345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.449253    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:56.481608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.481571    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"]
Nov 03 20:12:56.488131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.488099    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.506392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.503775    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2092ba-openshift-default-dockercfg-779gz\""
Nov 03 20:12:56.507782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.507755    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"]
Nov 03 20:12:56.664260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.654986    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672253    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-socket\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672325    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-data\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672468    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-istio-token\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672517    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479p4\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-kube-api-access-479p4\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672597    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-certs\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672676    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be91e70c-072f-4fab-be70-d26878da83e0-istio-podinfo\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672709    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-credential-socket\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672811    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be91e70c-072f-4fab-be70-d26878da83e0-istiod-ca-cert\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.673494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.672895    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-envoy\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.777669    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-istio-token\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.777738    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479p4\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-kube-api-access-479p4\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.777920    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-certs\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.777988    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be91e70c-072f-4fab-be70-d26878da83e0-istio-podinfo\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.778023    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-credential-socket\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.778123    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be91e70c-072f-4fab-be70-d26878da83e0-istiod-ca-cert\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.778233    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-envoy\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.778421    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-socket\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.778471    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-data\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.788219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.779025    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-data\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.798987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.798950    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be91e70c-072f-4fab-be70-d26878da83e0-istiod-ca-cert\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.800452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.799757    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-credential-socket\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.804281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.799975    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-socket\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.816614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.816575    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-certs\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.821293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.817528    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-istio-token\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.841010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.840969    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479p4\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-kube-api-access-479p4\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.856858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.856682    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be91e70c-072f-4fab-be70-d26878da83e0-istio-podinfo\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:56.858266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:56.858209    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-envoy\") pod \"e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") " pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:57.118488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:57.118403    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:57.201216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:57.200255    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:57.457137 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:12:57.457106    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe91e70c_072f_4fab_be70_d26878da83e0.slice/crio-865aed29313fc2b351fa8961f451cca5121dd0a158a5a8cd05972fea92d5c17a WatchSource:0}: Error finding container 865aed29313fc2b351fa8961f451cca5121dd0a158a5a8cd05972fea92d5c17a: Status 404 returned error can't find the container with id 865aed29313fc2b351fa8961f451cca5121dd0a158a5a8cd05972fea92d5c17a
Nov 03 20:12:57.460321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:57.460259    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:57.460416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:57.460354    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:57.460416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:57.460383    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:12:57.463486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:57.463464    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"]
Nov 03 20:12:57.496435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:57.491958    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:58.025984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:58.025938    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" event={"ID":"be91e70c-072f-4fab-be70-d26878da83e0","Type":"ContainerStarted","Data":"11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4"}
Nov 03 20:12:58.026167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:58.026018    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" event={"ID":"be91e70c-072f-4fab-be70-d26878da83e0","Type":"ContainerStarted","Data":"865aed29313fc2b351fa8961f451cca5121dd0a158a5a8cd05972fea92d5c17a"}
Nov 03 20:12:58.051314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:58.051259    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podStartSLOduration=2.051245133 podStartE2EDuration="2.051245133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:58.04749901 +0000 UTC m=+1970.716700478" watchObservedRunningTime="2026-11-03 20:12:58.051245133 +0000 UTC m=+1970.720446600"
Nov 03 20:12:58.118577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:58.118551    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:58.124104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:58.124080    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:58.240160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:58.240038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:59.078219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:59.077136    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:59.078219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:59.078049    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:12:59.093222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:59.088493    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:12:59.578233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:12:59.575807    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:00.630943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:00.630895    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:00.824235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:00.822800    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:01.294930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.294845    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"]
Nov 03 20:13:01.295350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.295325    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy" containerID="cri-o://795e45efd1191c2cc6b1b4746224f3de0141420706c2c2302130a256dd40b941" gracePeriod=30
Nov 03 20:13:01.484882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.484856    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:13:01.587112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587011    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-envoy\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587131    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-token\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-token\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587159    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-certs\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587277    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-podinfo\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587385    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-credential-socket\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587444    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-certs" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:01.587563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587512    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-kube-api-access-7sqws\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-kube-api-access-7sqws\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587555    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c004d68-fd7d-4827-ab19-fed3800b9e07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c004d68-fd7d-4827-ab19-fed3800b9e07-istiod-ca-cert\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587633    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-data\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587707    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-socket\") pod \"3c004d68-fd7d-4827-ab19-fed3800b9e07\" (UID: \"3c004d68-fd7d-4827-ab19-fed3800b9e07\") "
Nov 03 20:13:01.587993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.587971    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-data" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:01.588127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.588011    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-credential-socket" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:01.588127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.588044    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c004d68-fd7d-4827-ab19-fed3800b9e07-istiod-ca-cert" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:01.589164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.588360    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-socket" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:01.589256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.589167    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c004d68-fd7d-4827-ab19-fed3800b9e07-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.589256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.589192    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.589256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.589221    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.589256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.589236    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.589256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.589251    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.589772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.589749    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-envoy" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:01.589891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.589874    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-token" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:01.590932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.590909    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-kube-api-access-7sqws" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "kube-api-access-7sqws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:01.591338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.591317    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-podinfo" pod "3c004d68-fd7d-4827-ab19-fed3800b9e07" (UID: "3c004d68-fd7d-4827-ab19-fed3800b9e07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:13:01.616076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.616033    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:01.617391 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:01.617360    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:13:01.696223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.694422    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.696223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.694464    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sqws\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-kube-api-access-7sqws\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.696223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.694481    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.696223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.694495    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c004d68-fd7d-4827-ab19-fed3800b9e07-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:01.799262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.799231    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:13:01.813568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.813538    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:13:01.917261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.917134    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:01.979211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.979169    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"]
Nov 03 20:13:01.980129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.980104    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerName="istio-proxy"
Nov 03 20:13:01.980129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.980128    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerName="istio-proxy"
Nov 03 20:13:01.980375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.980151    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerName="istio-proxy"
Nov 03 20:13:01.980375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.980158    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerName="istio-proxy"
Nov 03 20:13:01.980482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.980433    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerName="istio-proxy"
Nov 03 20:13:01.980482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.980453    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerName="istio-proxy"
Nov 03 20:13:01.984733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.984714    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:01.990600 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.990577    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"]
Nov 03 20:13:01.991065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.990999    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2092ba\""
Nov 03 20:13:01.991187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.991172    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2092ba-dockercfg-csbvg\""
Nov 03 20:13:01.999499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.999470    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwc6\" (UniqueName: \"kubernetes.io/projected/358e261d-eb63-493c-bc15-8c202ee4af7f-kube-api-access-svwc6\") pod \"payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:01.999625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:01.999593    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e261d-eb63-493c-bc15-8c202ee4af7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:02.023316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.023288    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"]
Nov 03 20:13:02.028155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.028135    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.040797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.040755    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"]
Nov 03 20:13:02.081260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.079748    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:02.114217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.113174    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwc6\" (UniqueName: \"kubernetes.io/projected/358e261d-eb63-493c-bc15-8c202ee4af7f-kube-api-access-svwc6\") pod \"payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:02.114217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.113389    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e261d-eb63-493c-bc15-8c202ee4af7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:02.144439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.133561    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e261d-eb63-493c-bc15-8c202ee4af7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:02.234293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.233569    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwc6\" (UniqueName: \"kubernetes.io/projected/358e261d-eb63-493c-bc15-8c202ee4af7f-kube-api-access-svwc6\") pod \"payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:02.246440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.241600    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp24p\" (UniqueName: \"kubernetes.io/projected/7b4103a6-0edc-448e-b349-7679568bf8d2-kube-api-access-mp24p\") pod \"payload-processing-e2e-inf-a-2092ba-645468456d-bstpt\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.305222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.284745    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4103a6-0edc-448e-b349-7679568bf8d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2092ba-645468456d-bstpt\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.343238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.306567    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:02.446225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.427518    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp24p\" (UniqueName: \"kubernetes.io/projected/7b4103a6-0edc-448e-b349-7679568bf8d2-kube-api-access-mp24p\") pod \"payload-processing-e2e-inf-a-2092ba-645468456d-bstpt\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.446225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.434684    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4103a6-0edc-448e-b349-7679568bf8d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2092ba-645468456d-bstpt\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.455230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.450546    5282 generic.go:356] "Generic (PLEG): container finished" podID="3c004d68-fd7d-4827-ab19-fed3800b9e07" containerID="6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9" exitCode=0
Nov 03 20:13:02.481220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.456401    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4103a6-0edc-448e-b349-7679568bf8d2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2092ba-645468456d-bstpt\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.501225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.495416    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" event={"ID":"3c004d68-fd7d-4827-ab19-fed3800b9e07","Type":"ContainerDied","Data":"6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9"}
Nov 03 20:13:02.501225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.495532    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q" event={"ID":"3c004d68-fd7d-4827-ab19-fed3800b9e07","Type":"ContainerDied","Data":"9c17c6e12f2e2f751d5a096076a36282d201949e762065ceab35d9b103d63622"}
Nov 03 20:13:02.501225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.495564    5282 scope.go:122] "RemoveContainer" containerID="6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9"
Nov 03 20:13:02.501225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.495792    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"
Nov 03 20:13:02.552337 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.548781    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp24p\" (UniqueName: \"kubernetes.io/projected/7b4103a6-0edc-448e-b349-7679568bf8d2-kube-api-access-mp24p\") pod \"payload-processing-e2e-inf-a-2092ba-645468456d-bstpt\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.624142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.623901    5282 scope.go:122] "RemoveContainer" containerID="523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a"
Nov 03 20:13:02.636331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.636258    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"]
Nov 03 20:13:02.638324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.637042    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:13:02.641100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.641075    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5b918c5d-openshift-default-697bf5d6df-lgs6q"]
Nov 03 20:13:02.646993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.646944    5282 scope.go:122] "RemoveContainer" containerID="6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9"
Nov 03 20:13:02.647405 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:02.647378    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9\": container with ID starting with 6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9 not found: ID does not exist" containerID="6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9"
Nov 03 20:13:02.647478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.647419    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9"} err="failed to get container status \"6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9\": rpc error: code = NotFound desc = could not find container \"6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9\": container with ID starting with 6d62428521a0d09f59cbb0c78b779c788d85c62a315a86218b99175089f380c9 not found: ID does not exist"
Nov 03 20:13:02.647478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.647445    5282 scope.go:122] "RemoveContainer" containerID="523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a"
Nov 03 20:13:02.647742 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:02.647722    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a\": container with ID starting with 523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a not found: ID does not exist" containerID="523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a"
Nov 03 20:13:02.647780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.647749    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a"} err="failed to get container status \"523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a\": rpc error: code = NotFound desc = could not find container \"523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a\": container with ID starting with 523db8464ea3b17540fed4d793d790cd3ca6170f8e2f94e4eaf67f576d89358a not found: ID does not exist"
Nov 03 20:13:02.649160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.649141    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:02.719071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:02.718865    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:03.103118 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:13:03.103092    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358e261d_eb63_493c_bc15_8c202ee4af7f.slice/crio-850d0759fcc845605f7469d0e5eff6f960af8939223b1036b4c6a9f03d5e4aaa WatchSource:0}: Error finding container 850d0759fcc845605f7469d0e5eff6f960af8939223b1036b4c6a9f03d5e4aaa: Status 404 returned error can't find the container with id 850d0759fcc845605f7469d0e5eff6f960af8939223b1036b4c6a9f03d5e4aaa
Nov 03 20:13:03.103402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.103378    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"]
Nov 03 20:13:03.186318 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:13:03.186293    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4103a6_0edc_448e_b349_7679568bf8d2.slice/crio-21c1914f6d097aa4027294c641cfb7dbbcabd078941fba23b82a493a363119b3 WatchSource:0}: Error finding container 21c1914f6d097aa4027294c641cfb7dbbcabd078941fba23b82a493a363119b3: Status 404 returned error can't find the container with id 21c1914f6d097aa4027294c641cfb7dbbcabd078941fba23b82a493a363119b3
Nov 03 20:13:03.188363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.188335    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"]
Nov 03 20:13:03.405216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.405123    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:03.612532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.612462    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c004d68-fd7d-4827-ab19-fed3800b9e07" path="/var/lib/kubelet/pods/3c004d68-fd7d-4827-ab19-fed3800b9e07/volumes"
Nov 03 20:13:03.640318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.640267    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" event={"ID":"7b4103a6-0edc-448e-b349-7679568bf8d2","Type":"ContainerStarted","Data":"ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546"}
Nov 03 20:13:03.640587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.640565    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" event={"ID":"7b4103a6-0edc-448e-b349-7679568bf8d2","Type":"ContainerStarted","Data":"21c1914f6d097aa4027294c641cfb7dbbcabd078941fba23b82a493a363119b3"}
Nov 03 20:13:03.641760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.641743    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:03.644805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.644711    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" event={"ID":"358e261d-eb63-493c-bc15-8c202ee4af7f","Type":"ContainerStarted","Data":"c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6"}
Nov 03 20:13:03.644805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.644760    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" event={"ID":"358e261d-eb63-493c-bc15-8c202ee4af7f","Type":"ContainerStarted","Data":"850d0759fcc845605f7469d0e5eff6f960af8939223b1036b4c6a9f03d5e4aaa"}
Nov 03 20:13:03.644974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.644843    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:03.668344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.668181    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" podStartSLOduration=1.6681628640000001 podStartE2EDuration="1.668162864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:03.66425769 +0000 UTC m=+1976.333459171" watchObservedRunningTime="2026-11-03 20:13:03.668162864 +0000 UTC m=+1976.337364366"
Nov 03 20:13:03.696959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:03.696910    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" podStartSLOduration=2.696896889 podStartE2EDuration="2.696896889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:03.69210937 +0000 UTC m=+1976.361310838" watchObservedRunningTime="2026-11-03 20:13:03.696896889 +0000 UTC m=+1976.366098359"
Nov 03 20:13:04.240277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:04.238912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:04.873162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:04.873129    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:05.581897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:05.581855    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:06.356568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:06.355102    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:06.727898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:06.727868    5282 generic.go:356] "Generic (PLEG): container finished" podID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerID="795e45efd1191c2cc6b1b4746224f3de0141420706c2c2302130a256dd40b941" exitCode=0
Nov 03 20:13:06.728375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:06.727937    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" event={"ID":"954ebdee-c21e-404a-a2f3-6b43b0319060","Type":"ContainerDied","Data":"795e45efd1191c2cc6b1b4746224f3de0141420706c2c2302130a256dd40b941"}
Nov 03 20:13:06.983406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:06.983347    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:13:07.008179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.008139    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:07.088276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088227    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954ebdee-c21e-404a-a2f3-6b43b0319060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954ebdee-c21e-404a-a2f3-6b43b0319060-istiod-ca-cert\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.088458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088353    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-token\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-token\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.088458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088426    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-credential-socket\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.088548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088473    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-certs\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.088638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088620    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-envoy\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.088711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088690    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-kube-api-access-x6v8s\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-kube-api-access-x6v8s\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.088770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088733    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-credential-socket" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:07.088770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088650    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954ebdee-c21e-404a-a2f3-6b43b0319060-istiod-ca-cert" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:07.088869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088783    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-certs" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:07.088869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088788    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-podinfo\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.089022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.088982    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-data\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.089106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.089073    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-socket\") pod \"954ebdee-c21e-404a-a2f3-6b43b0319060\" (UID: \"954ebdee-c21e-404a-a2f3-6b43b0319060\") "
Nov 03 20:13:07.089310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.089281    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-data" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:07.089665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.089601    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-socket" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:07.090158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.090136    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.090242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.090158    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.090242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.090173    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.090242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.090183    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.090242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.090191    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954ebdee-c21e-404a-a2f3-6b43b0319060-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.090923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.090903    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-token" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:07.091342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.091323    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-podinfo" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:13:07.091400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.091381    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-kube-api-access-x6v8s" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "kube-api-access-x6v8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:07.091674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.091660    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-envoy" pod "954ebdee-c21e-404a-a2f3-6b43b0319060" (UID: "954ebdee-c21e-404a-a2f3-6b43b0319060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:07.198218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.197046    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.198218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.197084    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6v8s\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-kube-api-access-x6v8s\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.198218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.197099    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.198218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.197112    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954ebdee-c21e-404a-a2f3-6b43b0319060-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:07.531508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.531470    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:07.745221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.736600    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:07.821272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.820533    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr" event={"ID":"954ebdee-c21e-404a-a2f3-6b43b0319060","Type":"ContainerDied","Data":"121dd77465081628f2ff6deb3f3f0763b9c00265a4b22a2fe3dc67f8fd89a8f9"}
Nov 03 20:13:07.821272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.820613    5282 scope.go:122] "RemoveContainer" containerID="795e45efd1191c2cc6b1b4746224f3de0141420706c2c2302130a256dd40b941"
Nov 03 20:13:07.821272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.820795    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"
Nov 03 20:13:07.883223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.876876    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:07.947973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.932500    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"]
Nov 03 20:13:07.984002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:07.981072    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-hkbkr"]
Nov 03 20:13:08.114267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:08.110948    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:08.244879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:08.242769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:08.272045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:08.272008    5282 scope.go:122] "RemoveContainer" containerID="8c972fcb4fa9de6ac1d2afeb24cd7299a3abe27ce2259b46b7f651554cdceb9a"
Nov 03 20:13:08.366050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:08.365990    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:08.478253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:08.476863    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"]
Nov 03 20:13:08.478253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:08.477151    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy" containerID="cri-o://49eec8224f9ca146062d29d3fb56def448f393edf0346f9a33a1b82d8c49d6a1" gracePeriod=30
Nov 03 20:13:08.526210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:08.522455    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:09.458405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:09.458368    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" path="/var/lib/kubelet/pods/954ebdee-c21e-404a-a2f3-6b43b0319060/volumes"
Nov 03 20:13:09.611890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:09.611856    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:10.390286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:10.390258    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:11.450949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:11.450912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:12.301726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:12.301691    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:12.771687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:12.768903    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:13.137073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:13.136985    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:13.524366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:13.496753    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:13.925273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:13.919268    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"]
Nov 03 20:13:13.925273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:13.921139    5282 cpu_manager.go:414] "containerMap: removing container" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy"
Nov 03 20:13:13.925273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:13.921162    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy"
Nov 03 20:13:13.925273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:13.921444    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="954ebdee-c21e-404a-a2f3-6b43b0319060" containerName="istio-proxy"
Nov 03 20:13:14.047753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.047717    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:14.442220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.422893    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:14.456185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.456120    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:14.486232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.483277    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"]
Nov 03 20:13:14.486232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.485639    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:14.512881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.506249    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-babea4-openshift-default-dockercfg-gw2g8\""
Nov 03 20:13:14.950356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.947249    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-credential-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:14.950356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.947520    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-envoy\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:14.952094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.952066    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-certs\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:14.968586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.968554    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istiod-ca-cert\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:14.969511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.969489    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-token\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:14.969813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.969798    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-data\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:14.970093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:14.970080    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.213328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.200494    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-podinfo\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.213328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.200623    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6v2l\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-kube-api-access-r6v2l\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.213328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.175306    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332296    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332453    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-podinfo\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332520    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6v2l\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-kube-api-access-r6v2l\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332571    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-credential-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332611    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-envoy\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332761    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-certs\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332792    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istiod-ca-cert\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.332962    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-token\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.333019    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-data\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.401261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.342771    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-credential-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.539219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.515360    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.550220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.542103    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-certs\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.573482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.573440    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istiod-ca-cert\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.596668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.596529    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:13:15.627224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.616983    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-data\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:15.666016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.625311    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:13:15.666397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.665458    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 03 20:13:15.724223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.722550    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:15.755122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.754862    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:15.937919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:15.908109    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.046229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.038655    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-token\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:16.105965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.097490    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:16.189955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.169715    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:13:16.189955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.173125    5282 generic.go:356] "Generic (PLEG): container finished" podID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerID="49eec8224f9ca146062d29d3fb56def448f393edf0346f9a33a1b82d8c49d6a1" exitCode=0
Nov 03 20:13:16.216122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.213860    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-podinfo\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:16.216122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.214089    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-envoy\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:16.216122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.214630    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6v2l\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-kube-api-access-r6v2l\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:16.256329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.254497    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.270254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.270219    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.320976    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-podinfo\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321090    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-data\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321136    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-envoy\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321258    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-kube-api-access-8qkvh\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-kube-api-access-8qkvh\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321349    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321380    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-token\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-token\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321430    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-socket\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321499    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-credential-socket\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.325502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.321546    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-certs\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.359655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.354491    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-data" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:16.390863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.378276    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-socket" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:16.397242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.396729    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-credential-socket" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:16.401242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.397946    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.434282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.425120    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-certs" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:16.434282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.432819    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:16.434282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.432851    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:16.434282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.432869    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:16.434282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.432882    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:16.471281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.461396    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.500829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.495851    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.548247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.525585    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.571281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.548840    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:16.659147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.645284    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\") pod \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\" (UID: \"f28e0e71-7687-4f07-a1e0-4d6382ac4d74\") "
Nov 03 20:13:16.669416 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:13:16.668551    5282 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f28e0e71-7687-4f07-a1e0-4d6382ac4d74/volumes/kubernetes.io~configmap/istiod-ca-cert
Nov 03 20:13:16.669416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.668577    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:16.669416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.668644    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.680232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.674543    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.681006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.679496    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.713841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.710731    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:16.766971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.758775    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.882238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.799302    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.882238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.817427    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:16.882238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.860279    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-envoy" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:13:16.882238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.862394    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.882238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.862665    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:13:16.882238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.863006    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-podinfo" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:13:16.917220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.915537    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:16.917220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.915575    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:16.917220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.916486    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-kube-api-access-8qkvh" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "kube-api-access-8qkvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:16.938549    5282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.491s"
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.938619    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" event={"ID":"f28e0e71-7687-4f07-a1e0-4d6382ac4d74","Type":"ContainerDied","Data":"49eec8224f9ca146062d29d3fb56def448f393edf0346f9a33a1b82d8c49d6a1"}
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.938748    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.938800    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.938839    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.938882    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.939008    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.939039    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:13:16.951952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:16.940172    5282 scope.go:122] "RemoveContainer" containerID="49eec8224f9ca146062d29d3fb56def448f393edf0346f9a33a1b82d8c49d6a1"
Nov 03 20:13:17.161799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.149272    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:13:17.161799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.161739    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-token" pod "f28e0e71-7687-4f07-a1e0-4d6382ac4d74" (UID: "f28e0e71-7687-4f07-a1e0-4d6382ac4d74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:17.236152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.171569    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qkvh\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-kube-api-access-8qkvh\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:17.236152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.171606    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28e0e71-7687-4f07-a1e0-4d6382ac4d74-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:17.236152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.209807    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:17.342298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.342156    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.695999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.548065    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:13:17.695999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.694011    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Nov 03 20:13:17.695999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.695979    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:13:17.835902 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.835861    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.997109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.884546    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.997109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.885048    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Nov 03 20:13:17.997109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.907407    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:13:17.997109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.907632    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.997109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.835751    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:17.997109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.913726    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:17.997109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:17.990837    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:13:18.031893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.016405    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:13:18.417908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.417803    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:13:18.449301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.443455    5282 scope.go:122] "RemoveContainer" containerID="0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767"
Nov 03 20:13:18.451784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.451754    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:18.685458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.635239    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:13:18.745054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.742374    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Nov 03 20:13:18.893286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.765413    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:18.936389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.936284    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:18.951217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:18.936599    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:19.063231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.058677    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": context deadline exceeded"
Nov 03 20:13:19.088476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.088260    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"
Nov 03 20:13:19.108898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.108852    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:19.143598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.129970    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:13:19.143598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.130333    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:13:19.143598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.130745    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:19.143598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.131015    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:13:19.156246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.146006    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:19.281885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.111109    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:19.281885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.111283    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Nov 03 20:13:19.281885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.111306    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:13:19.281885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.112113    5282 patch_prober.go:29] interesting pod/thanos-querier-5c8cd547dc-9csvs container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:13:19.281885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.241076    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-5c8cd547dc-9csvs" podUID="9a113235-00b3-42db-ac16-0fc01da5a788" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:19.337360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.319561    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:19.543307 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:19.543266    5282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.527s"
Nov 03 20:13:19.543758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.543324    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25" event={"ID":"f28e0e71-7687-4f07-a1e0-4d6382ac4d74","Type":"ContainerDied","Data":"f07e1bc4a352e64fd7faf2124c6096b59f4273887f5591b765b583269e09ea22"}
Nov 03 20:13:19.543758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.543460    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:13:19.556145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.555514    5282 scope.go:122] "RemoveContainer" containerID="0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767"
Nov 03 20:13:19.598218 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:19.570930    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767\": container with ID starting with 0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767 not found: ID does not exist" containerID="0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767"
Nov 03 20:13:19.598218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.570971    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767"} err="failed to get container status \"0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767\": rpc error: code = NotFound desc = could not find container \"0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767\": container with ID starting with 0b1be24ff4504cbff7b87f8017757aeebc55fe5855c42d47aaaef6f78fc70767 not found: ID does not exist"
Nov 03 20:13:19.598218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.571024    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:19.598218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.571158    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:19.598218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.571270    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:19.618216 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:19.617217    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28e0e71_7687_4f07_a1e0_4d6382ac4d74.slice/crio-f07e1bc4a352e64fd7faf2124c6096b59f4273887f5591b765b583269e09ea22\": RecentStats: unable to find data in memory cache]"
Nov 03 20:13:19.629685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.629655    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:13:19.629893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.629878    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:13:19.630014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.630000    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:13:19.630329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.630313    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:13:19.630479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.630465    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:13:19.630679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.630655    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:13:19.630842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.630826    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:13:19.631000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.630987    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:13:19.631128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.631113    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:13:19.631476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.631456    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:13:19.631700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.631680    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:13:19.632418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.632399    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:13:19.632630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.632619    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:13:19.632810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.632800    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:13:19.635755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.633175    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:13:19.635755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.635037    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:13:19.635755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.635149    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:13:19.635755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.635271    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:13:19.635755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.635400    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:13:19.637328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.637309    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:13:19.653684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.653639    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"]
Nov 03 20:13:19.718091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.718055    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:19.835505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.835435    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"]
Nov 03 20:13:19.843126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:19.843097    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-acd98d21-openshift-default-7ffbccb87c-99k25"]
Nov 03 20:13:20.136500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:20.134447    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" event={"ID":"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e","Type":"ContainerStarted","Data":"552f103a81a916d81f0053af9578427b1fb978148383d30d94b8bdc65941f452"}
Nov 03 20:13:20.136500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:20.134511    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" event={"ID":"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e","Type":"ContainerStarted","Data":"e48729caceecdf949843bc95b6eb857faa4853d9facfc4677f8d167bdbc8bddd"}
Nov 03 20:13:20.174229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:20.171733    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podStartSLOduration=7.171709314 podStartE2EDuration="7.171709314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:20.15996839 +0000 UTC m=+1992.829169862" watchObservedRunningTime="2026-11-03 20:13:20.171709314 +0000 UTC m=+1992.840910783"
Nov 03 20:13:20.329221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:20.324791    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:20.329221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:20.324921    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:13:20.329221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:20.324965    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:13:20.646281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:20.634068    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:21.226175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:21.226136    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:21.287802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:21.282114    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:21.425531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:21.425470    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:21.463313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:21.463277    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" path="/var/lib/kubelet/pods/f28e0e71-7687-4f07-a1e0-4d6382ac4d74/volumes"
Nov 03 20:13:21.579780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:21.579517    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:22.527817 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:22.505663    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:22.582221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:22.574046    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:13:22.618252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:22.593659    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:22.812853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:22.812735    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:22.959967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:22.959877    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:23.612221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:23.602734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:24.021059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.021016    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"]
Nov 03 20:13:24.022001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.021970    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy"
Nov 03 20:13:24.022001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.021991    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy"
Nov 03 20:13:24.022244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.022023    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy"
Nov 03 20:13:24.022244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.022029    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy"
Nov 03 20:13:24.022244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.022162    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy"
Nov 03 20:13:24.024822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.024801    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:24.031439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.031419    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.036914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.036891    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"]
Nov 03 20:13:24.042009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.041989    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-43b780\"/\"default-dockercfg-qxjl2\""
Nov 03 20:13:24.042520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.042469    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-43b780\"/\"kube-root-ca.crt\""
Nov 03 20:13:24.042656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.042587    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-43b780\"/\"test-model-43b780-kserve-self-signed-certs\""
Nov 03 20:13:24.042743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.042609    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-43b780\"/\"openshift-service-ca.crt\""
Nov 03 20:13:24.074189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.074154    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6v5z\" (UniqueName: \"kubernetes.io/projected/340a1e46-7aa9-423f-be41-4804a49a01d3-kube-api-access-x6v5z\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.074391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.074276    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-dshm\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.074631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.074602    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-home\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.074722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.074705    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-model-cache\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.074898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.074878    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-tmp-dir\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.075035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.074953    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a1e46-7aa9-423f-be41-4804a49a01d3-tls-certs\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.176417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.176380    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-tmp-dir\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.176633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.176468    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a1e46-7aa9-423f-be41-4804a49a01d3-tls-certs\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.176633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.176541    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6v5z\" (UniqueName: \"kubernetes.io/projected/340a1e46-7aa9-423f-be41-4804a49a01d3-kube-api-access-x6v5z\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.176633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.176612    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-dshm\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.176841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.176815    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-tmp-dir\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.176941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.176922    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-home\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.176994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.176979    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-model-cache\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.177367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.177339    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-home\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.177499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.177426    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-model-cache\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.179392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.179367    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a1e46-7aa9-423f-be41-4804a49a01d3-tls-certs\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.179512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.179411    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-dshm\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.184023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.183993    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6v5z\" (UniqueName: \"kubernetes.io/projected/340a1e46-7aa9-423f-be41-4804a49a01d3-kube-api-access-x6v5z\") pod \"test-model-43b780-kserve-5b68c65c59-lchnb\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") " pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.227131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.227094    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:24.362629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.362541    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:24.804850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:24.802939    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:25.197120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:25.196981    5282 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:13:25.197120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:25.197054    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:13:25.212682 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:13:25.212638    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340a1e46_7aa9_423f_be41_4804a49a01d3.slice/crio-5da2bfb9527aa76a7942c3d862ac9ab80db63ea8c577fe9bfb2764529706a022 WatchSource:0}: Error finding container 5da2bfb9527aa76a7942c3d862ac9ab80db63ea8c577fe9bfb2764529706a022: Status 404 returned error can't find the container with id 5da2bfb9527aa76a7942c3d862ac9ab80db63ea8c577fe9bfb2764529706a022
Nov 03 20:13:25.212984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:25.212958    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"]
Nov 03 20:13:25.687601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:25.687563    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" event={"ID":"340a1e46-7aa9-423f-be41-4804a49a01d3","Type":"ContainerStarted","Data":"7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985"}
Nov 03 20:13:25.687601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:25.687614    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" event={"ID":"340a1e46-7aa9-423f-be41-4804a49a01d3","Type":"ContainerStarted","Data":"5da2bfb9527aa76a7942c3d862ac9ab80db63ea8c577fe9bfb2764529706a022"}
Nov 03 20:13:25.710345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:25.710290    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" podStartSLOduration=2.7102752150000002 podStartE2EDuration="2.710275215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:25.705720705 +0000 UTC m=+1998.374922174" watchObservedRunningTime="2026-11-03 20:13:25.710275215 +0000 UTC m=+1998.379476741"
Nov 03 20:13:25.742966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:25.742924    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:26.365267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:26.365228    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:26.598436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:26.598404    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"]
Nov 03 20:13:26.612247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:26.612214    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"]
Nov 03 20:13:26.631681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:26.631580    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:26.779992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:26.779958    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:27.442836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:27.442805    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:27.611716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:27.611263    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:28.277720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:28.277685    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:28.882316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:28.882173    5282 generic.go:356] "Generic (PLEG): container finished" podID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerID="d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38" exitCode=0
Nov 03 20:13:28.882846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:28.882537    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" event={"ID":"0f42d2e4-8389-474b-9c77-eb1fd340edad","Type":"ContainerDied","Data":"d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38"}
Nov 03 20:13:28.883251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:28.883178    5282 scope.go:122] "RemoveContainer" containerID="d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38"
Nov 03 20:13:28.885386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:28.885353    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:28.885469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:28.885429    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:28.885469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:28.885459    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:29.028949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:29.028894    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:29.790093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:29.790046    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:29.969221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:29.964517    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" event={"ID":"0f42d2e4-8389-474b-9c77-eb1fd340edad","Type":"ContainerStarted","Data":"730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c"}
Nov 03 20:13:30.599016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:30.598841    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:30.743765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:30.738614    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:13:30.743765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:30.739400    5282 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:13:30.743765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:30.739461    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:13:31.369038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:31.368998    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:31.601335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:31.601277    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x" podUID="61953a05-7db9-4b00-97ce-1cb92090b97b" containerName="payload-processing" containerID="cri-o://dcf4e38093fd7b20c03923fd666c1bb83556060d4f60d9945c577df8eae6d818" gracePeriod=25
Nov 03 20:13:31.623315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:31.623233    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg" podUID="30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" containerName="payload-pre-processing" containerID="cri-o://f7668fe421c3cb4d688f365ba8a6d65835e5267313cfc7d104d438117e41b8a5" gracePeriod=25
Nov 03 20:13:31.657243 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:31.657083    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:13:31.735239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:31.735125    5282 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:13:31.735428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:31.735250    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:13:32.023236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.023182    5282 generic.go:356] "Generic (PLEG): container finished" podID="30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" containerID="f7668fe421c3cb4d688f365ba8a6d65835e5267313cfc7d104d438117e41b8a5" exitCode=0
Nov 03 20:13:32.023463 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.023443    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg" event={"ID":"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1","Type":"ContainerDied","Data":"f7668fe421c3cb4d688f365ba8a6d65835e5267313cfc7d104d438117e41b8a5"}
Nov 03 20:13:32.025739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.025709    5282 generic.go:356] "Generic (PLEG): container finished" podID="61953a05-7db9-4b00-97ce-1cb92090b97b" containerID="dcf4e38093fd7b20c03923fd666c1bb83556060d4f60d9945c577df8eae6d818" exitCode=0
Nov 03 20:13:32.025904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.025883    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x" event={"ID":"61953a05-7db9-4b00-97ce-1cb92090b97b","Type":"ContainerDied","Data":"dcf4e38093fd7b20c03923fd666c1bb83556060d4f60d9945c577df8eae6d818"}
Nov 03 20:13:32.112086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.112057    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:13:32.116780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.116754    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:13:32.145892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.139869    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:32.237220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.233892    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61953a05-7db9-4b00-97ce-1cb92090b97b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61953a05-7db9-4b00-97ce-1cb92090b97b-plugins-config-volume\") pod \"61953a05-7db9-4b00-97ce-1cb92090b97b\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") "
Nov 03 20:13:32.237220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.234037    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-plugins-config-volume\") pod \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") "
Nov 03 20:13:32.237220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.234076    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-kube-api-access-fzznq\" (UniqueName: \"kubernetes.io/projected/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-kube-api-access-fzznq\") pod \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\" (UID: \"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1\") "
Nov 03 20:13:32.237220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.234157    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61953a05-7db9-4b00-97ce-1cb92090b97b-kube-api-access-sk247\" (UniqueName: \"kubernetes.io/projected/61953a05-7db9-4b00-97ce-1cb92090b97b-kube-api-access-sk247\") pod \"61953a05-7db9-4b00-97ce-1cb92090b97b\" (UID: \"61953a05-7db9-4b00-97ce-1cb92090b97b\") "
Nov 03 20:13:32.291063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.282415    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-plugins-config-volume" pod "30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" (UID: "30bb5035-4ea1-4fe1-82e7-f8eea347bbb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:32.291670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.291471    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61953a05-7db9-4b00-97ce-1cb92090b97b-plugins-config-volume" pod "61953a05-7db9-4b00-97ce-1cb92090b97b" (UID: "61953a05-7db9-4b00-97ce-1cb92090b97b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:32.305278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.304401    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-kube-api-access-fzznq" pod "30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" (UID: "30bb5035-4ea1-4fe1-82e7-f8eea347bbb1"). InnerVolumeSpecName "kube-api-access-fzznq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:32.308062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.308013    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61953a05-7db9-4b00-97ce-1cb92090b97b-kube-api-access-sk247" pod "61953a05-7db9-4b00-97ce-1cb92090b97b" (UID: "61953a05-7db9-4b00-97ce-1cb92090b97b"). InnerVolumeSpecName "kube-api-access-sk247". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:32.349349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.349236    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61953a05-7db9-4b00-97ce-1cb92090b97b-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.349349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.349276    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.349349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.349293    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzznq\" (UniqueName: \"kubernetes.io/projected/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1-kube-api-access-fzznq\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.349349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.349308    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk247\" (UniqueName: \"kubernetes.io/projected/61953a05-7db9-4b00-97ce-1cb92090b97b-kube-api-access-sk247\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:32.734169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.734130    5282 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:13:32.735118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.734753    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:13:32.801754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:32.801717    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:33.036057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.036026    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"
Nov 03 20:13:33.036252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.036181    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg" event={"ID":"30bb5035-4ea1-4fe1-82e7-f8eea347bbb1","Type":"ContainerDied","Data":"9db9b7f9f7a38397434642a4692408c5cd6bb35d536ad6294bbf3b9dfe51da91"}
Nov 03 20:13:33.036336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.036251    5282 scope.go:122] "RemoveContainer" containerID="f7668fe421c3cb4d688f365ba8a6d65835e5267313cfc7d104d438117e41b8a5"
Nov 03 20:13:33.040803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.038147    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"
Nov 03 20:13:33.040803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.038155    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x" event={"ID":"61953a05-7db9-4b00-97ce-1cb92090b97b","Type":"ContainerDied","Data":"17838213f07f4c4391bfb13b8fe6a3eae43f3f4b867bf688bc84d73d56324804"}
Nov 03 20:13:33.048992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.048975    5282 scope.go:122] "RemoveContainer" containerID="dcf4e38093fd7b20c03923fd666c1bb83556060d4f60d9945c577df8eae6d818"
Nov 03 20:13:33.062452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.062426    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"]
Nov 03 20:13:33.070740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.070719    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-caa01a-7f7dccf9bc-69rpg"]
Nov 03 20:13:33.113582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.113538    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"]
Nov 03 20:13:33.117208 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.117157    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-caa01a-5f66f584db-gx57x"]
Nov 03 20:13:33.462348 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.462262    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" path="/var/lib/kubelet/pods/30bb5035-4ea1-4fe1-82e7-f8eea347bbb1/volumes"
Nov 03 20:13:33.462915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.462898    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61953a05-7db9-4b00-97ce-1cb92090b97b" path="/var/lib/kubelet/pods/61953a05-7db9-4b00-97ce-1cb92090b97b/volumes"
Nov 03 20:13:33.495613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.495579    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:33.734073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.734042    5282 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:13:33.734256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:33.734094    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:13:34.115494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:34.115370    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:34.364824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:34.364790    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:34.406552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:34.406478    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:34.734046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:34.734013    5282 patch_prober.go:29] interesting pod/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 20:13:34.734232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:34.734080    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 20:13:34.955870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:34.955820    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:35.160618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:35.160530    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:13:35.697882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:35.697850    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:35.955467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:35.955386    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:13:35.956660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:35.956636    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:13:35.957753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:35.957733    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:13:36.490731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:36.490698    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:37.427986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:37.427945    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:38.204604 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:38.204568    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:39.369189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:39.369114    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:39.562966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:39.559428    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:39.937918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:39.937885    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:40.791814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:40.791782    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:41.096298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:41.095213    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:41.577721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:41.577678    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:41.876465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:41.876342    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:42.231908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:42.231856    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:42.431238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:42.431180    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:43.062008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:43.061970    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:43.851144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:43.851110    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:44.615690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:44.615656    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:45.480153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:45.480122    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:46.113514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:46.113474    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:46.255672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:46.255637    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:46.409331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:46.402294    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:47.223272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.223193    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:47.407648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.407540    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:47.409184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.409156    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"]
Nov 03 20:13:47.410655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.410633    5282 cpu_manager.go:414] "containerMap: removing container" podUID="61953a05-7db9-4b00-97ce-1cb92090b97b" containerName="payload-processing"
Nov 03 20:13:47.410823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.410807    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61953a05-7db9-4b00-97ce-1cb92090b97b" containerName="payload-processing"
Nov 03 20:13:47.410938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.410926    5282 cpu_manager.go:414] "containerMap: removing container" podUID="30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" containerName="payload-pre-processing"
Nov 03 20:13:47.411024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.411014    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" containerName="payload-pre-processing"
Nov 03 20:13:47.411432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.411412    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="61953a05-7db9-4b00-97ce-1cb92090b97b" containerName="payload-processing"
Nov 03 20:13:47.411549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.411537    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28e0e71-7687-4f07-a1e0-4d6382ac4d74" containerName="istio-proxy"
Nov 03 20:13:47.411634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.411624    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="30bb5035-4ea1-4fe1-82e7-f8eea347bbb1" containerName="payload-pre-processing"
Nov 03 20:13:47.429620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.429575    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"]
Nov 03 20:13:47.429792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.429644    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"]
Nov 03 20:13:47.429879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.429587    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:47.435988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.435945    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9b1aed\""
Nov 03 20:13:47.436288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.436235    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:47.438846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.437263    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9b1aed-dockercfg-w5ds4\""
Nov 03 20:13:47.444112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.443422    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"]
Nov 03 20:13:47.750274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.712870    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:47.750274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.727699    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd9726a-f255-4fe9-b09b-b1aa20027e09-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:47.750274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.727963    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfvq\" (UniqueName: \"kubernetes.io/projected/bfd9726a-f255-4fe9-b09b-b1aa20027e09-kube-api-access-clfvq\") pod \"payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:47.750274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.728061    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbcq\" (UniqueName: \"kubernetes.io/projected/60502a0d-1c95-4527-a002-7b71716a397b-kube-api-access-4mbcq\") pod \"payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:47.750274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.728121    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60502a0d-1c95-4527-a002-7b71716a397b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:47.848251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.833558    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60502a0d-1c95-4527-a002-7b71716a397b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:47.848251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.833742    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd9726a-f255-4fe9-b09b-b1aa20027e09-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:47.848251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.833936    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfvq\" (UniqueName: \"kubernetes.io/projected/bfd9726a-f255-4fe9-b09b-b1aa20027e09-kube-api-access-clfvq\") pod \"payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:47.848251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.834019    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbcq\" (UniqueName: \"kubernetes.io/projected/60502a0d-1c95-4527-a002-7b71716a397b-kube-api-access-4mbcq\") pod \"payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:47.874246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.851524    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd9726a-f255-4fe9-b09b-b1aa20027e09-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:47.944681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:47.944632    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfvq\" (UniqueName: \"kubernetes.io/projected/bfd9726a-f255-4fe9-b09b-b1aa20027e09-kube-api-access-clfvq\") pod \"payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:48.020190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.019996    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60502a0d-1c95-4527-a002-7b71716a397b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:48.051263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.026187    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbcq\" (UniqueName: \"kubernetes.io/projected/60502a0d-1c95-4527-a002-7b71716a397b-kube-api-access-4mbcq\") pod \"payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:48.065610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.065579    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:48.127308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.127274    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:48.291280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.291149    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:48.445037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.444997    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"]
Nov 03 20:13:48.467740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.467704    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"]
Nov 03 20:13:48.489044 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:13:48.489010    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd9726a_f255_4fe9_b09b_b1aa20027e09.slice/crio-138cdf38efbba7ca53b8553ff5b43d34335b849d6e7c9cfd8047cb7b5d7745a3 WatchSource:0}: Error finding container 138cdf38efbba7ca53b8553ff5b43d34335b849d6e7c9cfd8047cb7b5d7745a3: Status 404 returned error can't find the container with id 138cdf38efbba7ca53b8553ff5b43d34335b849d6e7c9cfd8047cb7b5d7745a3
Nov 03 20:13:48.885217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.885119    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" event={"ID":"60502a0d-1c95-4527-a002-7b71716a397b","Type":"ContainerStarted","Data":"90de7e0df5d90bc10e398881b372726463a34d5137943892ece604e9f047e691"}
Nov 03 20:13:48.885385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.885188    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:48.885385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.885243    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" event={"ID":"60502a0d-1c95-4527-a002-7b71716a397b","Type":"ContainerStarted","Data":"19ee746af4cf3f06a06df277996668e67bb5b5a7da009641174499cc52cc1b3a"}
Nov 03 20:13:48.886834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.886808    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" event={"ID":"bfd9726a-f255-4fe9-b09b-b1aa20027e09","Type":"ContainerStarted","Data":"709907f46bf9b4f0a46b55e7f130ccc1706b5bc2bf660ca19678c19b615b23b7"}
Nov 03 20:13:48.887030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.886849    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" event={"ID":"bfd9726a-f255-4fe9-b09b-b1aa20027e09","Type":"ContainerStarted","Data":"138cdf38efbba7ca53b8553ff5b43d34335b849d6e7c9cfd8047cb7b5d7745a3"}
Nov 03 20:13:48.887030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.886886    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:13:48.907345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.907288    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" podStartSLOduration=1.907269377 podStartE2EDuration="1.907269377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:48.900119102 +0000 UTC m=+2021.569320572" watchObservedRunningTime="2026-11-03 20:13:48.907269377 +0000 UTC m=+2021.576470848"
Nov 03 20:13:48.908163 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.908136    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:48.919782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:48.919735    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" podStartSLOduration=1.9197207459999999 podStartE2EDuration="1.919720746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:48.918340652 +0000 UTC m=+2021.587542123" watchObservedRunningTime="2026-11-03 20:13:48.919720746 +0000 UTC m=+2021.588922208"
Nov 03 20:13:49.610557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:49.610523    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:50.290025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:50.289981    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:51.006437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:51.006404    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:51.803358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:51.803324    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:52.402184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:52.402149    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:52.851383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:52.851351    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:52.994265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:52.994186    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:53.183005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.182924    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"]
Nov 03 20:13:53.191408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.190190    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.201271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.201234    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-22a567-openshift-default-dockercfg-w792x\""
Nov 03 20:13:53.204127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.204099    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"]
Nov 03 20:13:53.247335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.247297    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-credential-socket\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.247584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.247566    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a030b91-7598-43b4-8554-ba348887cf30-istio-podinfo\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.247772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.247755    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a030b91-7598-43b4-8554-ba348887cf30-istiod-ca-cert\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.247932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.247917    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-envoy\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.248047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.248032    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-socket\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.248157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.248065    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbfm\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-kube-api-access-xnbfm\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.248157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.248143    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-data\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.248378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.248174    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-certs\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.248378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.248231    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-istio-token\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.248880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.248859    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:53.487945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.487902    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-envoy\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.487989    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-socket\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.488024    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbfm\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-kube-api-access-xnbfm\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.488229    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-data\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.488273    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-certs\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.488316    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-istio-token\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.488391    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-credential-socket\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.488465    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a030b91-7598-43b4-8554-ba348887cf30-istio-podinfo\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.488790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.488647    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a030b91-7598-43b4-8554-ba348887cf30-istiod-ca-cert\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.504261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.489582    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a030b91-7598-43b4-8554-ba348887cf30-istiod-ca-cert\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.504261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.491497    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-data\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.504261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.498564    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-socket\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.504261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.499317    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-certs\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.504261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.499566    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-credential-socket\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.580864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.580827    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-istio-token\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.643605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.642594    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbfm\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-kube-api-access-xnbfm\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.702431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.702391    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:53.703471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.703433    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-envoy\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.711276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.709259    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a030b91-7598-43b4-8554-ba348887cf30-istio-podinfo\") pod \"e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") " pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:53.838298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:53.824884    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:54.093831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:54.093666    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:54.317174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:54.317119    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"]
Nov 03 20:13:54.350160 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:13:54.350113    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a030b91_7598_43b4_8554_ba348887cf30.slice/crio-853f18ee3b52e6aae1a6ef5a407becaf87e9156478117aee241c425d32afd140 WatchSource:0}: Error finding container 853f18ee3b52e6aae1a6ef5a407becaf87e9156478117aee241c425d32afd140: Status 404 returned error can't find the container with id 853f18ee3b52e6aae1a6ef5a407becaf87e9156478117aee241c425d32afd140
Nov 03 20:13:54.353990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:54.353945    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:54.354109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:54.354040    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:54.354109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:54.354087    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:13:54.773777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:54.773741    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:55.012794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.012755    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" event={"ID":"5a030b91-7598-43b4-8554-ba348887cf30","Type":"ContainerStarted","Data":"7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22"}
Nov 03 20:13:55.013006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.012812    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" event={"ID":"5a030b91-7598-43b4-8554-ba348887cf30","Type":"ContainerStarted","Data":"853f18ee3b52e6aae1a6ef5a407becaf87e9156478117aee241c425d32afd140"}
Nov 03 20:13:55.038379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.038279    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podStartSLOduration=2.038264043 podStartE2EDuration="2.038264043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:55.033384491 +0000 UTC m=+2027.702585963" watchObservedRunningTime="2026-11-03 20:13:55.038264043 +0000 UTC m=+2027.707465513"
Nov 03 20:13:55.491760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.491730    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:55.826616 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.826532    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:55.832524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.832497    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:55.953992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.953953    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"]
Nov 03 20:13:55.954639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:55.954607    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz" podUID="09a6dcfe-9e9a-47ab-aa1d-7f295178c637" containerName="maas-api" containerID="cri-o://e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4" gracePeriod=30
Nov 03 20:13:56.022009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.021980    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:56.023330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.023305    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:13:56.025712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.025683    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:56.066415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.066376    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz" podUID="09a6dcfe-9e9a-47ab-aa1d-7f295178c637" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.130:8443/health\": dial tcp 10.133.0.130:8443: connect: connection refused"
Nov 03 20:13:56.277297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.277076    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:56.619219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.616231    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:56.659090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.659060    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:13:56.707219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.701451    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-openshift-service-ca\") pod \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") "
Nov 03 20:13:56.707219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.701507    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-maas-api-tls\") pod \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") "
Nov 03 20:13:56.707219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.703586    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-openshift-service-ca" pod "09a6dcfe-9e9a-47ab-aa1d-7f295178c637" (UID: "09a6dcfe-9e9a-47ab-aa1d-7f295178c637"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:56.716019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.715934    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-maas-api-tls" pod "09a6dcfe-9e9a-47ab-aa1d-7f295178c637" (UID: "09a6dcfe-9e9a-47ab-aa1d-7f295178c637"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:13:56.719082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.718016    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:56.719082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.718051    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:56.794915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.794885    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:56.838293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.819690    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-metrics-certs\") pod \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") "
Nov 03 20:13:56.838293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.819816    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-kube-api-access-j4z9f\" (UniqueName: \"kubernetes.io/projected/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-kube-api-access-j4z9f\") pod \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\" (UID: \"09a6dcfe-9e9a-47ab-aa1d-7f295178c637\") "
Nov 03 20:13:56.882499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.879608    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-metrics-certs" pod "09a6dcfe-9e9a-47ab-aa1d-7f295178c637" (UID: "09a6dcfe-9e9a-47ab-aa1d-7f295178c637"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:13:56.897086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.887787    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-kube-api-access-j4z9f" pod "09a6dcfe-9e9a-47ab-aa1d-7f295178c637" (UID: "09a6dcfe-9e9a-47ab-aa1d-7f295178c637"). InnerVolumeSpecName "kube-api-access-j4z9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:56.926217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.925690    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:56.926217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:56.925728    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4z9f\" (UniqueName: \"kubernetes.io/projected/09a6dcfe-9e9a-47ab-aa1d-7f295178c637-kube-api-access-j4z9f\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.034723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.034683    5282 generic.go:356] "Generic (PLEG): container finished" podID="09a6dcfe-9e9a-47ab-aa1d-7f295178c637" containerID="e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4" exitCode=0
Nov 03 20:13:57.034914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.034754    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"
Nov 03 20:13:57.034914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.034775    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz" event={"ID":"09a6dcfe-9e9a-47ab-aa1d-7f295178c637","Type":"ContainerDied","Data":"e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4"}
Nov 03 20:13:57.034914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.034828    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz" event={"ID":"09a6dcfe-9e9a-47ab-aa1d-7f295178c637","Type":"ContainerDied","Data":"7721f31622150f2332402a194ebf04107c7aa28bdc7b0403ab7928a0f4db64c8"}
Nov 03 20:13:57.034914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.034852    5282 scope.go:122] "RemoveContainer" containerID="e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4"
Nov 03 20:13:57.048077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.048059    5282 scope.go:122] "RemoveContainer" containerID="e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4"
Nov 03 20:13:57.048422 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:13:57.048402    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4\": container with ID starting with e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4 not found: ID does not exist" containerID="e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4"
Nov 03 20:13:57.048502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.048430    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4"} err="failed to get container status \"e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4\": rpc error: code = NotFound desc = could not find container \"e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4\": container with ID starting with e9517aaf106a3b5199e3aaeedfa9418bb3fc10b532653d1794f827a2aa7ee8b4 not found: ID does not exist"
Nov 03 20:13:57.061299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.061271    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"]
Nov 03 20:13:57.065982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.065956    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-caa01a-5d9bbdc85-4r5jz"]
Nov 03 20:13:57.163816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.163739    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:57.459649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.459556    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a6dcfe-9e9a-47ab-aa1d-7f295178c637" path="/var/lib/kubelet/pods/09a6dcfe-9e9a-47ab-aa1d-7f295178c637/volumes"
Nov 03 20:13:57.832324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.827506    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:57.971084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:57.971049    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:59.240394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.236709    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:59.498163 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.498086    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:59.570908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.570872    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"]
Nov 03 20:13:59.572245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.572221    5282 cpu_manager.go:414] "containerMap: removing container" podUID="09a6dcfe-9e9a-47ab-aa1d-7f295178c637" containerName="maas-api"
Nov 03 20:13:59.572245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.572242    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a6dcfe-9e9a-47ab-aa1d-7f295178c637" containerName="maas-api"
Nov 03 20:13:59.572456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.572442    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a6dcfe-9e9a-47ab-aa1d-7f295178c637" containerName="maas-api"
Nov 03 20:13:59.581560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.581538    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.590369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.590341    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"]
Nov 03 20:13:59.689442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.667810    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-22a567-dockercfg-cmljz\""
Nov 03 20:13:59.708841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.690828    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-22a567\""
Nov 03 20:13:59.708841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.692606    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:13:59.708841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.695529    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jx97\" (UniqueName: \"kubernetes.io/projected/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-kube-api-access-8jx97\") pod \"payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.708841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.695891    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.848354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.833414    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jx97\" (UniqueName: \"kubernetes.io/projected/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-kube-api-access-8jx97\") pod \"payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.848354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.833651    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.848354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.834843    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.874051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.874011    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jx97\" (UniqueName: \"kubernetes.io/projected/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-kube-api-access-8jx97\") pod \"payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.923336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.917932    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:13:59.944933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.936411    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:13:59.966221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:13:59.958375    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:14:00.420408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:00.420357    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:00.594733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:00.594703    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"]
Nov 03 20:14:00.597310 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:14:00.597283    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a6d7c7_3fe1_45dd_a4ff_2b66c0547cac.slice/crio-8ae9027a20271e6e7896f203633edaaef9622b2d117eb30c7625856a9c8f6f2c WatchSource:0}: Error finding container 8ae9027a20271e6e7896f203633edaaef9622b2d117eb30c7625856a9c8f6f2c: Status 404 returned error can't find the container with id 8ae9027a20271e6e7896f203633edaaef9622b2d117eb30c7625856a9c8f6f2c
Nov 03 20:14:01.218557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:01.218523    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:01.421248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:01.421179    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" event={"ID":"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac","Type":"ContainerStarted","Data":"72a6485d80472dea967a4cc228bffdd476cf5f446a5f34841be9fe5049bf8f60"}
Nov 03 20:14:01.421716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:01.421279    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" event={"ID":"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac","Type":"ContainerStarted","Data":"8ae9027a20271e6e7896f203633edaaef9622b2d117eb30c7625856a9c8f6f2c"}
Nov 03 20:14:01.421716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:01.421528    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:14:01.442455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:01.442408    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" podStartSLOduration=2.442393053 podStartE2EDuration="2.442393053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:01.440902224 +0000 UTC m=+2034.110103695" watchObservedRunningTime="2026-11-03 20:14:01.442393053 +0000 UTC m=+2034.111594522"
Nov 03 20:14:01.727990 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:14:01.727949    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:14:02.009077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:02.008992    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:02.674218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:02.673824    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:03.237116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.237082    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"]
Nov 03 20:14:03.242381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.242362    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.246125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.246087    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9b1aed-serving-cert\""
Nov 03 20:14:03.253470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.253446    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"]
Nov 03 20:14:03.291121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.291092    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.291323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.291159    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788mm\" (UniqueName: \"kubernetes.io/projected/deaba6ad-8109-48f9-81ca-62e226deff03-kube-api-access-788mm\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.291382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.291356    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-metrics-certs\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.291580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.291568    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaba6ad-8109-48f9-81ca-62e226deff03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.362732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.362705    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:03.408218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.406565    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-metrics-certs\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.408218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.406958    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaba6ad-8109-48f9-81ca-62e226deff03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.408218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.407065    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.408218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.407387    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788mm\" (UniqueName: \"kubernetes.io/projected/deaba6ad-8109-48f9-81ca-62e226deff03-kube-api-access-788mm\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.419219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.413623    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaba6ad-8109-48f9-81ca-62e226deff03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.484256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.461681    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788mm\" (UniqueName: \"kubernetes.io/projected/deaba6ad-8109-48f9-81ca-62e226deff03-kube-api-access-788mm\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.547279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.524490    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:03.547279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.525277    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.547279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.541762    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-metrics-certs\") pod \"maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.559818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.559542    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:03.664860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:03.664828    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:04.411221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:04.407419    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.039564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.016693    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.219356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.217703    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"]
Nov 03 20:14:05.221326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.221297    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:14:05.266218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.262136    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.396851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.396804    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.955169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.949453    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"]
Nov 03 20:14:05.955791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.955668    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:05.959800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.958646    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:05.964124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.963763    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-663013\"/\"default-dockercfg-6p7r5\""
Nov 03 20:14:05.964124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.963828    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-663013\"/\"test-model-663013-kserve-self-signed-certs\""
Nov 03 20:14:05.967588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.965952    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-663013\"/\"openshift-service-ca.crt\""
Nov 03 20:14:05.967588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.966018    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-663013\"/\"kube-root-ca.crt\""
Nov 03 20:14:05.984373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:05.984324    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"]
Nov 03 20:14:06.136221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.113680    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtlnb\" (UniqueName: \"kubernetes.io/projected/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-kube-api-access-wtlnb\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.136221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.113727    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tls-certs\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.136221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.113886    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-model-cache\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.136221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.114007    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tmp-dir\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.136221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.114032    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-home\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.136221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.114079    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-dshm\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.205256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.197693    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:06.207664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.206183    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6" event={"ID":"deaba6ad-8109-48f9-81ca-62e226deff03","Type":"ContainerStarted","Data":"edf9547ceb1addd7007254e8422223a1ed37fc4119ee42cb6c578527e334cc0a"}
Nov 03 20:14:06.233776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.217834    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlnb\" (UniqueName: \"kubernetes.io/projected/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-kube-api-access-wtlnb\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.233776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.217889    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tls-certs\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.233776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.218057    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-model-cache\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.233776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.218218    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tmp-dir\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.233776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.218263    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-home\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.233776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.218326    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-dshm\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.233776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.224186    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-model-cache\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.235999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.235451    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tmp-dir\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.235999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.235756    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-home\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.292278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.289533    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tls-certs\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.292757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.292693    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-dshm\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.305243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.302043    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtlnb\" (UniqueName: \"kubernetes.io/projected/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-kube-api-access-wtlnb\") pod \"test-model-663013-kserve-884b9f4f-zsnh6\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") " pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.465838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.465795    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:06.481733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.481703    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:14:06.598293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.595339    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:06.729315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.729190    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:06.954278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:06.914298    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:07.244667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:07.244641    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"]
Nov 03 20:14:07.270625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:07.268770    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6" event={"ID":"deaba6ad-8109-48f9-81ca-62e226deff03","Type":"ContainerStarted","Data":"d1a93542c7e4a9d6031960099b6b96aeae36beacb0c4049cace18211b78d4e9b"}
Nov 03 20:14:07.270625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:07.268862    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:07.308232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:07.295659    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6" podStartSLOduration=3.717975526 podStartE2EDuration="4.295637618s" totalImagesPullingTime="577.662092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:07.294656961 +0000 UTC m=+2039.963858432" watchObservedRunningTime="2026-11-03 20:14:07.295637618 +0000 UTC m=+2039.964839090"
Nov 03 20:14:07.505402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:07.505190    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:08.357268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:08.357225    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" event={"ID":"3b558ab4-c66b-48b8-a93d-c8f9a48211dd","Type":"ContainerStarted","Data":"27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375"}
Nov 03 20:14:08.357910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:08.357883    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" event={"ID":"3b558ab4-c66b-48b8-a93d-c8f9a48211dd","Type":"ContainerStarted","Data":"451b948fb86ed723115727e0a18c0d2738b8ffaa1a893d9162fc4377bbe77d46"}
Nov 03 20:14:08.428105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:08.428061    5282 scope.go:122] "RemoveContainer" containerID="3d6497c18a03b9c133b881a480397b9cfdd9460b6e80d2eb85e454880e0c00a8"
Nov 03 20:14:08.442492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:08.442469    5282 scope.go:122] "RemoveContainer" containerID="9be12bf7b96c80d5af59771f6862bf6360914475f9504dea973e8c53b29b51ed"
Nov 03 20:14:08.609273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:08.599347    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:09.264222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:09.263347    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:09.686271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:09.680576    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:14:09.686271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:09.680972    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:14:09.748271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:09.747376    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:09.782224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:09.769650    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:09.792402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:09.790553    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:14:09.792402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:09.790887    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:14:10.225772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:10.225736    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:10.513813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:10.513376    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:10.687324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:10.687280    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:10.688478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:10.688452    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:14:10.741574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:10.741503    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" podStartSLOduration=5.741479008 podStartE2EDuration="5.741479008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:08.411628861 +0000 UTC m=+2041.080830332" watchObservedRunningTime="2026-11-03 20:14:10.741479008 +0000 UTC m=+2043.410680478"
Nov 03 20:14:11.016228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.009936    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:11.106296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.104473    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:11.384833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.384024    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:11.524236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.499129    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:11.524236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.499533    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Nov 03 20:14:11.533108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.533069    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:14:11.597221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.571838    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:14:11.818158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.578889    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:11.892749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.864561    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:11.892749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.865545    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:14:11.892749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.866339    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Nov 03 20:14:11.953251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.942698    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:14:11.967338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:11.657597    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:14:12.667481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.044125    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:12.891773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.865801    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:12.891773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.577857    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:14:12.891773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.866466    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:14:12.891773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.866568    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:12.891773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.717127    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.145:8000/ready\": context deadline exceeded"
Nov 03 20:14:12.891773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.785630    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:12.892595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.891850    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:14:12.892595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.891945    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:14:13.016697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.016029    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.083043    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.050297    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.053552    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.053835    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b" podUID="95213a91-c3dd-4195-bcea-9e329e10679b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.083219    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:12.169482    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.190428    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl" podUID="04e563bf-735c-4d94-9de1-d04994490f4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.198351    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.198444    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.198532    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.198590    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:14:13.199774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.023971    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 03 20:14:13.282354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.206556    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.299551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.283429    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 20:14:13.460076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.460030    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Nov 03 20:14:13.611276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.531075    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.649490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.621787    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 03 20:14:13.672234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.375387    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Nov 03 20:14:13.695380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.693777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:14:13.695380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.693876    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:14:13.745744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.745708    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:13.747439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.746945    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.748026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.747996    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Nov 03 20:14:13.863402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.863293    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.866367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.866327    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:13.866733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.866709    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:14:13.939435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.740026    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:14:13.939435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.926362    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:14:13.939435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.926424    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:14:13.939435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.926460    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:14:13.981329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:13.981278    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:14.193906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.193806    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:14.197120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.197067    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 03 20:14:14.199033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.198977    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:14:14.199531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.199509    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:14:14.199703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.199689    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:14:14.359264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.236482    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 03 20:14:14.359264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.322142    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:14.472297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.457987    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:14.514222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.503935    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:14:14.676232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.657649    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:14:14.676232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.668482    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:14:14.676232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.669518    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:14:14.707682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.707642    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:14.733282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.707910    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Nov 03 20:14:14.771481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.771439    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:14.917223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.893283    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:14:14.917223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.908603    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:14.951094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:14.951065    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:14:15.050441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.050340    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:15.069226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.056037    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:14:15.069226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.056103    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:14:15.091392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.073647    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:14:15.198351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.198319    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:14:15.242584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.242549    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:15.243753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.243729    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:14:15.249250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.244674    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:14:15.249250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.246780    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:14:15.258167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.258146    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:14:15.269747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.263972    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:14:15.269747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.267464    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:14:15.271434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:15.271417    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:14:16.297222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:16.294326    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:16.453833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:16.453772    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:16.620314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:16.603312    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:16.996288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:16.977181    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:17.520443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.520337    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:17.591429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.591396    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"]
Nov 03 20:14:17.610314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.603979    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:17.612879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.612850    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:14:17.617336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.617307    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"]
Nov 03 20:14:17.625064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.624946    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:17.625664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.625413    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznlf\" (UniqueName: \"kubernetes.io/projected/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-kube-api-access-qznlf\") pod \"payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:17.693342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.667772    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:17.767221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.766543    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2c6c50-dockercfg-lvqfr\""
Nov 03 20:14:17.767221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.766673    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2c6c50\""
Nov 03 20:14:17.894303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:17.812562    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:18.022050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:18.022019    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznlf\" (UniqueName: \"kubernetes.io/projected/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-kube-api-access-qznlf\") pod \"payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:18.034097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:18.019755    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:18.194815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:18.175830    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznlf\" (UniqueName: \"kubernetes.io/projected/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-kube-api-access-qznlf\") pod \"payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:18.233876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:18.233557    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:18.541185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:18.541147    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"]
Nov 03 20:14:18.542474 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:14:18.542443    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa329b0a_2fe4_4229_bbde_c5dc6a0ebcea.slice/crio-87c04818ba2cf1b33cc70248d4852e731a9833f76fa93e21f7616b952d03618a WatchSource:0}: Error finding container 87c04818ba2cf1b33cc70248d4852e731a9833f76fa93e21f7616b952d03618a: Status 404 returned error can't find the container with id 87c04818ba2cf1b33cc70248d4852e731a9833f76fa93e21f7616b952d03618a
Nov 03 20:14:18.911098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:18.911017    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:19.073742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:19.073710    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n" event={"ID":"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea","Type":"ContainerStarted","Data":"a2b5fd22d41516800ad9f9cdc01c33a6d523a6ba1b3e13209bd004055b2c5cf3"}
Nov 03 20:14:19.073951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:19.073772    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n" event={"ID":"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea","Type":"ContainerStarted","Data":"87c04818ba2cf1b33cc70248d4852e731a9833f76fa93e21f7616b952d03618a"}
Nov 03 20:14:19.073951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:19.073802    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:19.101396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:19.101343    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n" podStartSLOduration=2.101327161 podStartE2EDuration="2.101327161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:19.098630373 +0000 UTC m=+2051.767831841" watchObservedRunningTime="2026-11-03 20:14:19.101327161 +0000 UTC m=+2051.770528636"
Nov 03 20:14:19.928268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:19.927993    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:20.643427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:20.642331    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:14:20.643427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:20.642395    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:20.719510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:20.719469    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:20.870112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:20.868421    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:21.066963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:21.066927    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:21.263402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:21.263306    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:21.803001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:21.802962    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:22.691919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:22.691870    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:23.310933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:23.310895    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:23.433710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:23.433334    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": read tcp 10.133.0.2:40304->10.133.0.144:15021: read: connection reset by peer" start-of-body=
Nov 03 20:14:23.433710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:23.433398    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": read tcp 10.133.0.2:40304->10.133.0.144:15021: read: connection reset by peer"
Nov 03 20:14:24.069890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.069858    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:24.204111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.204078    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:24.461035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.460942    5282 generic.go:356] "Generic (PLEG): container finished" podID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerID="552f103a81a916d81f0053af9578427b1fb978148383d30d94b8bdc65941f452" exitCode=0
Nov 03 20:14:24.461504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.461468    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" event={"ID":"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e","Type":"ContainerDied","Data":"552f103a81a916d81f0053af9578427b1fb978148383d30d94b8bdc65941f452"}
Nov 03 20:14:24.471227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.462189    5282 scope.go:122] "RemoveContainer" containerID="552f103a81a916d81f0053af9578427b1fb978148383d30d94b8bdc65941f452"
Nov 03 20:14:24.473093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.469126    5282 generic.go:356] "Generic (PLEG): container finished" podID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerID="2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172" exitCode=0
Nov 03 20:14:24.473093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.469151    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" event={"ID":"60336029-d74c-4227-a77f-7f2bfc2a17f1","Type":"ContainerDied","Data":"2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172"}
Nov 03 20:14:24.473283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.473096    5282 scope.go:122] "RemoveContainer" containerID="2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172"
Nov 03 20:14:24.492331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.492287    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:24.492610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.492580    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:24.492799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.492765    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:24.493162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.493127    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:24.493280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.493233    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:24.493351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.493284    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:14:24.880297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:24.880242    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:25.483285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:25.483246    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" event={"ID":"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e","Type":"ContainerStarted","Data":"c9283a2bc3eb8cabacbe72d21ee521d2ba00c6c87ffd0c5de27a9b57c4020b3d"}
Nov 03 20:14:25.485777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:25.485743    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" event={"ID":"60336029-d74c-4227-a77f-7f2bfc2a17f1","Type":"ContainerStarted","Data":"504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878"}
Nov 03 20:14:25.502042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:25.502014    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:14:25.502252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:25.502191    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:14:25.502319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:25.502246    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:14:25.510492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:25.510457    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:26.271374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:26.271342    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:14:26.271578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:26.271522    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:14:26.271659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:26.271584    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:14:26.319222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:26.318695    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:26.520650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:26.520602    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:14:26.536145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:26.521234    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:14:27.280246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:27.278702    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:14:27.280246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:27.278782    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:14:27.295253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:27.293833    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:27.506708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:27.503953    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:14:27.506708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:27.504022    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:14:28.291222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:28.285790    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:14:28.291222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:28.285860    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:14:28.304219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:28.304020    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:28.505234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:28.503408    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:14:28.505234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:28.503485    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:14:29.013223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:29.000115    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:29.286283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:29.285368    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:14:29.286283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:29.285428    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:14:29.509224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:29.508928    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:14:29.509224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:29.508987    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:14:29.727368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:29.727294    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:30.125303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:30.125189    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:14:30.271664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:30.271573    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:14:30.271950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:30.271916    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:14:30.406283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:30.405966    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:30.517217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:30.513475    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:14:30.517217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:30.513545    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:14:31.134221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:31.117156    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:31.295639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:31.295593    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Nov 03 20:14:31.295935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:31.295906    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Nov 03 20:14:31.506375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:31.506337    5282 patch_prober.go:29] interesting pod/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 20:14:31.521771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:31.521623    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 20:14:31.658612 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:14:31.658582    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:14:31.763547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:31.763466    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:31.905221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:31.889835    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:32.326535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:32.326494    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:14:32.328033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:32.328001    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:14:32.328883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:32.328859    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:14:32.635886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:32.635799    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:32.755221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:32.754364    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:14:32.755758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:32.755733    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:14:32.770220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:32.768833    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:14:34.085954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:34.085916    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:34.201660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:34.201625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:34.812683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:34.812651    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:34.955274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:34.954018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:35.112139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:35.112051    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:35.810089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:35.810053    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:36.379294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:36.379259    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:36.694750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:36.694618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:37.436561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:37.436525    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:38.187594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:38.187557    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:38.936683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:38.936643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:39.726729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:39.726699    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:40.471116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:40.471069    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:41.255039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:41.254998    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:41.916317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:41.916284    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:42.627089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:42.627051    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:43.413465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:43.413432    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:44.185162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:44.185132    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:44.871341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:44.871305    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:45.611888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:45.611857    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:46.291523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:46.291489    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:47.001932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:47.001891    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:47.716313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:47.716275    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:48.482245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:48.480319    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:49.162778 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:49.162682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:49.721084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:49.721051    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:49.855218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:49.854346    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:50.501359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:50.501325    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:51.236229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:51.235314    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:52.065219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:52.062837    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:52.681651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:52.681609    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:53.284142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:53.284109    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:54.085246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:54.085164    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:55.084692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:55.084657    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:56.559658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:56.559616    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:57.460009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:57.459904    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:58.234968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:58.234931    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:59.004380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:59.004343    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:14:59.871320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:14:59.871289    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:00.797222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:00.796738    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:01.611621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:01.611589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:01.694611 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:15:01.694568    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:15:02.358721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:02.358689    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:02.990216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:02.990171    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:03.876582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:03.876544    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:04.482148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:04.482113    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:05.191549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:05.191506    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:05.869618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:05.869580    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:06.602829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:06.602790    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:07.402822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:07.402773    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:08.263333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:08.262421    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:08.712446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:08.712356    5282 generic.go:356] "Generic (PLEG): container finished" podID="5a030b91-7598-43b4-8554-ba348887cf30" containerID="7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22" exitCode=0
Nov 03 20:15:08.712924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:08.712437    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" event={"ID":"5a030b91-7598-43b4-8554-ba348887cf30","Type":"ContainerDied","Data":"7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22"}
Nov 03 20:15:08.712988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:08.712976    5282 scope.go:122] "RemoveContainer" containerID="7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22"
Nov 03 20:15:08.714953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:08.714921    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:08.715077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:08.715012    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:08.715077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:08.715062    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:09.180984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:09.180942    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:09.724399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:09.724358    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" event={"ID":"5a030b91-7598-43b4-8554-ba348887cf30","Type":"ContainerStarted","Data":"5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26"}
Nov 03 20:15:09.826135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:09.826099    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:15:09.827297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:09.827270    5282 patch_prober.go:29] interesting pod/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:15:09.827441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:09.827328    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:15:09.976284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:09.972122    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:10.826781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:10.826749    5282 patch_prober.go:29] interesting pod/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:15:10.827176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:10.826807    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:15:10.927123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:10.927088    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:11.025225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:11.023892    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:15:11.620276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:11.620237    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:11.826885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:11.826848    5282 patch_prober.go:29] interesting pod/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:15:11.827500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:11.827469    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:15:11.854838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:11.854809    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:11.996510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:11.996477    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:12.382942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:12.382850    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:12.827030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:12.826990    5282 patch_prober.go:29] interesting pod/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:15:12.827559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:12.827048    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:15:13.118394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:13.118282    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:13.798267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:13.798111    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:13.826772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:13.826733    5282 patch_prober.go:29] interesting pod/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:15:13.827055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:13.827024    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:15:14.416502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:14.416446    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:14.826052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:14.826015    5282 patch_prober.go:29] interesting pod/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:15:14.826231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:14.826084    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 20:15:15.118076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:15.117984    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:15.858278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:15.857422    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:15.884080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:15.884049    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:15:15.927571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:15.927480    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:15:16.488646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:16.488615    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:17.588097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:17.587927    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:18.486862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:18.486818    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:19.154367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:19.154333    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:19.406070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:19.405954    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:19.605049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:19.605015    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:20.035135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:20.035101    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:20.231775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:20.231734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:21.240259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:21.240219    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:21.390220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:21.385490    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:21.445107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:21.443878    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:15:22.401500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:22.401468    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:23.100209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:23.100167    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:24.346242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:24.344661    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": context deadline exceeded"
Nov 03 20:15:24.456793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:24.456759    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:25.420056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:25.420025    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:26.310723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:26.310688    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:27.042354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:27.042236    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:27.842655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:27.842621    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:28.629265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:28.629186    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:29.388267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:29.387095    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:30.002924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:30.002880    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:30.722094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:30.722058    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:31.463894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:31.463850    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:31.619308 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:15:31.619275    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:15:32.086239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:32.067287    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:32.834241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:32.834188    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:33.595957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:33.595923    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:33.742519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:33.742426    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:34.187791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:34.187755    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:34.316160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:34.316116    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:34.499122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:34.499080    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:35.275257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:35.275141    5282 generic.go:356] "Generic (PLEG): container finished" podID="be91e70c-072f-4fab-be70-d26878da83e0" containerID="11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4" exitCode=0
Nov 03 20:15:35.275257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:35.275235    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" event={"ID":"be91e70c-072f-4fab-be70-d26878da83e0","Type":"ContainerDied","Data":"11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4"}
Nov 03 20:15:35.275744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:35.275715    5282 scope.go:122] "RemoveContainer" containerID="11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4"
Nov 03 20:15:35.277846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:35.277813    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:35.277946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:35.277890    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:35.277946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:35.277928    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:15:35.500172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:35.500143    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:36.199748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:36.199709    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:36.360027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:36.359983    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" event={"ID":"be91e70c-072f-4fab-be70-d26878da83e0","Type":"ContainerStarted","Data":"02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088"}
Nov 03 20:15:36.817263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:36.817190    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:37.119234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:37.119141    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:15:37.119396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:37.119334    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:15:37.119396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:37.119384    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:15:37.656296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:37.656229    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:38.118970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:38.118939    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:15:38.119165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:38.119002    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:15:39.119528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:39.119496    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:15:39.119930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:39.119554    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:15:40.119300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:40.119243    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:15:40.119484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:40.119324    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:15:40.901154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:40.901116    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:41.119577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:41.119063    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 03 20:15:41.119577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:41.119133    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 03 20:15:41.541525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:41.541489    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:42.146206 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:42.146160    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:15:42.146982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:42.146955    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:15:42.149142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:42.149120    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:15:42.206320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:42.206274    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:42.955900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:42.955862    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:43.819803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:43.819768    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:44.452392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:44.452337    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:45.086871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:45.086838    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:45.851920 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:45.851879    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:46.534108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:46.520722    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:47.122759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:47.122697    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:47.900652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:47.900610    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:48.658230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:48.657273    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:49.280897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:49.280864    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:49.981999 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:49.981965    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:50.683714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:50.683674    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:51.401185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:51.401146    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:52.193618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:52.193583    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:52.823062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:52.823027    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:53.597608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:53.597537    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:54.305074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:54.305040    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:55.083192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:55.083132    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:55.893134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:55.893105    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:56.674095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:56.674059    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:57.356481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:57.356444    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:57.978404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:57.978370    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:58.675107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:58.675072    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:15:59.397522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:15:59.397479    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:00.131232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:00.130820    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:00.785342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:00.785308    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:01.515539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:01.515511    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:01.697569 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:16:01.697540    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:16:02.198832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:02.198794    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:02.809683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:02.809648    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:03.520230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:03.520164    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:04.177566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:04.177528    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:04.794995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:04.794961    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:05.485668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:05.485639    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:06.111588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:06.111548    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:06.780541 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:06.780512    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:07.380727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:07.380692    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:08.014795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:08.014764    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:08.696505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:08.696476    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:08.862809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:08.862766    5282 scope.go:122] "RemoveContainer" containerID="7bf066e4780e2bb76172a1232f40f7d39f026a62c62666c22164b08c167faa87"
Nov 03 20:16:09.320693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:09.320659    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:10.017623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:10.017590    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:10.767006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:10.766922    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:11.292168 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:11.292105    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:11.909001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:11.908968    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:12.596548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:12.596517    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:13.285205 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:13.285168    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:13.889051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:13.889001    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:14.680922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:14.680871    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:15.271813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:15.271752    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:15.872381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:15.872333    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:16.688622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:16.688571    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:17.401923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:17.401883    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:18.038807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:18.038765    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:18.740274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:18.740223    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:19.485501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:19.485437    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:20.075681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:20.075647    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:20.791410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:20.791367    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:21.358237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:21.358174    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:21.499872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:21.499844    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:22.160687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:22.160643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:22.826454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:22.826417    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:23.573227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:23.566953    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:24.196489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:24.196454    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:25.089445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:25.089408    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:25.945803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:25.945768    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:26.700228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:26.700182    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:27.402803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:27.402769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:27.986211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:27.986167    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:28.646420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:28.646386    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:29.354766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:29.354721    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:29.892437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:29.892403    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:30.497431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:30.497393    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:31.222036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:31.222003    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:31.619652 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:16:31.619620    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:16:31.899688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:31.899612    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:32.575895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:32.575835    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:33.241144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:33.241111    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:33.904939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:33.904902    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:34.498638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:34.498600    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:35.106136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:35.106059    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:35.781138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:35.781104    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:36.554828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:36.554789    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:37.153862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:37.153827    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:37.878927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:37.878889    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:38.233180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:38.233148    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:16:38.652751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:38.652654    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:39.412225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:39.408841    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:40.392238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:40.388827    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:41.416221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:41.415306    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kpkjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:41.416221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:41.415387    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:41.613222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:41.612555    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:42.511373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:42.511343    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:43.182070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:43.182036    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:43.861207 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:43.861161    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:44.618699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:44.618661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:45.274379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:45.274342    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:45.884649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:45.884604    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:46.515985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:46.515951    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:47.198075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:47.198039    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:47.799652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:47.799614    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:48.508609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:48.508574    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:49.417275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:49.413233    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:50.055737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:50.055702    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:50.680890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:50.680856    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:51.396166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:51.396133    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:52.080125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:52.080092    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:52.679119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:52.679086    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:53.300419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:53.300387    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:53.903878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:53.903846    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:54.580919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:54.580886    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:55.285165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:55.285134    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:56.014978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:56.014934    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:56.700842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:56.700805    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:57.287492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:57.287460    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:57.974007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:57.973974    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:58.659216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:58.658797    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:16:59.281135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:16:59.281096    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:00.015693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:00.015655    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:00.843237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:00.843177    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:01.514385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:01.514344    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:01.694683 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:17:01.694648    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:17:02.287147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:02.287113    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:02.880944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:02.880910    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:03.501570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:03.501528    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:04.153280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:04.153172    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:04.914223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:04.905732    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:05.588925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:05.588889    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:06.285687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:06.285653    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:07.071038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:07.071007    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:07.717188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:07.717145    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:08.385861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:08.385832    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:08.940094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:08.940064    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"]
Nov 03 20:17:08.940347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:08.940277    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy" containerID="cri-o://30fc5d65e6909017c91360c2b41069b6b07ed343dde60ecb5c8946badcbc126d" gracePeriod=30
Nov 03 20:17:09.118410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:09.118351    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:09.903802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:09.903767    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:10.534132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:10.534092    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:11.194113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:11.194085    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:11.982940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:11.982906    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:12.594070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:12.594040    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:13.201685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:13.201654    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:13.886578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:13.886544    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:14.170880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.170793    5282 generic.go:356] "Generic (PLEG): container finished" podID="0b10504a-edeb-4638-86aa-52e54efea88b" containerID="30fc5d65e6909017c91360c2b41069b6b07ed343dde60ecb5c8946badcbc126d" exitCode=0
Nov 03 20:17:14.171141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.171116    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" event={"ID":"0b10504a-edeb-4638-86aa-52e54efea88b","Type":"ContainerDied","Data":"30fc5d65e6909017c91360c2b41069b6b07ed343dde60ecb5c8946badcbc126d"}
Nov 03 20:17:14.252703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.252676    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:17:14.284592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.284560    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"]
Nov 03 20:17:14.285722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.285701    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy"
Nov 03 20:17:14.285795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.285724    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy"
Nov 03 20:17:14.285897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.285888    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" containerName="istio-proxy"
Nov 03 20:17:14.289889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.289867    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.304124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.303262    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"]
Nov 03 20:17:14.336922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.336677    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-data\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.336922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.336758    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-credential-socket\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.336922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.336876    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-istio-token\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-istio-token\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.337477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.337007    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b10504a-edeb-4638-86aa-52e54efea88b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b10504a-edeb-4638-86aa-52e54efea88b-istiod-ca-cert\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.337477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.337171    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-certs\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.337477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.337240    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b10504a-edeb-4638-86aa-52e54efea88b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b10504a-edeb-4638-86aa-52e54efea88b-istio-podinfo\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.337477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.337386    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-credential-socket" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.337477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.337412    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-envoy\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.337841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.337550    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-kube-api-access-gbv7j\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-kube-api-access-gbv7j\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.337841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.337695    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-socket\") pod \"0b10504a-edeb-4638-86aa-52e54efea88b\" (UID: \"0b10504a-edeb-4638-86aa-52e54efea88b\") "
Nov 03 20:17:14.338278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.338253    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b10504a-edeb-4638-86aa-52e54efea88b-istiod-ca-cert" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:14.338476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.338456    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-data" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.339015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.338993    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.339160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.339145    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.339292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.339276    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b10504a-edeb-4638-86aa-52e54efea88b-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.339402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.338535    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-socket" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.339547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.339518    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-certs" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.341666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.341586    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b10504a-edeb-4638-86aa-52e54efea88b-istio-podinfo" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:14.342652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.342556    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-kube-api-access-gbv7j" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "kube-api-access-gbv7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:14.342652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.342542    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-envoy" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:14.344239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.344174    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-istio-token" pod "0b10504a-edeb-4638-86aa-52e54efea88b" (UID: "0b10504a-edeb-4638-86aa-52e54efea88b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:14.441381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441300    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84qm4\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-kube-api-access-84qm4\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441342    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-credential-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441492    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-podinfo\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441547    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441659    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-token\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441778 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441766    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istiod-ca-cert\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441912    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-certs\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441943    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-data\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.441996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.441982    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-envoy\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.442112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.442102    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.442145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.442115    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.442145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.442125    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b10504a-edeb-4638-86aa-52e54efea88b-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.442145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.442134    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.442318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.442144    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbv7j\" (UniqueName: \"kubernetes.io/projected/0b10504a-edeb-4638-86aa-52e54efea88b-kube-api-access-gbv7j\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.442318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.442154    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b10504a-edeb-4638-86aa-52e54efea88b-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:14.544041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544002    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-podinfo\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544049    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544117    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-token\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544223    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istiod-ca-cert\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544543    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544558    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-certs\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544616    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-data\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544690    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-envoy\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544759    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84qm4\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-kube-api-access-84qm4\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544792    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-credential-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544869    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-certs\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.544942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544914    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-data\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.545057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.544973    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istiod-ca-cert\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.545350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.545318    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-credential-socket\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.546823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.546801    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-podinfo\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.546910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.546845    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-envoy\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.552429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.552402    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-token\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.552602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.552586    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84qm4\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-kube-api-access-84qm4\") pod \"e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") " pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:14.566709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.566686    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:14.622285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:14.621242    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:15.055765 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:17:15.055733    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5fa49b_6181_408e_87a3_3a63ae7b4a74.slice/crio-fa6335a910a5cb2c4b66d36315ebc2ee12f7c2a1ce17dae16478dffdcf23fc2c WatchSource:0}: Error finding container fa6335a910a5cb2c4b66d36315ebc2ee12f7c2a1ce17dae16478dffdcf23fc2c: Status 404 returned error can't find the container with id fa6335a910a5cb2c4b66d36315ebc2ee12f7c2a1ce17dae16478dffdcf23fc2c
Nov 03 20:17:15.056705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.056683    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"]
Nov 03 20:17:15.059175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.059140    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:15.059295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.059218    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:15.059295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.059252    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:15.187058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.187021    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547" event={"ID":"0b10504a-edeb-4638-86aa-52e54efea88b","Type":"ContainerDied","Data":"e6ae26a5efc324f2f6996376a98ce92da5b667d1c00766b6dad3349e8516df43"}
Nov 03 20:17:15.187058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.187052    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"
Nov 03 20:17:15.187324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.187094    5282 scope.go:122] "RemoveContainer" containerID="30fc5d65e6909017c91360c2b41069b6b07ed343dde60ecb5c8946badcbc126d"
Nov 03 20:17:15.190053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.190022    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" event={"ID":"9b5fa49b-6181-408e-87a3-3a63ae7b4a74","Type":"ContainerStarted","Data":"23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f"}
Nov 03 20:17:15.190215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.190079    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" event={"ID":"9b5fa49b-6181-408e-87a3-3a63ae7b4a74","Type":"ContainerStarted","Data":"fa6335a910a5cb2c4b66d36315ebc2ee12f7c2a1ce17dae16478dffdcf23fc2c"}
Nov 03 20:17:15.213442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.213396    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" podStartSLOduration=106.213382726 podStartE2EDuration="1m46.213382726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:15.212631706 +0000 UTC m=+2227.881833170" watchObservedRunningTime="2026-11-03 20:17:15.213382726 +0000 UTC m=+2227.882584195"
Nov 03 20:17:15.229598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.229571    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"]
Nov 03 20:17:15.237683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.237653    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-g6547"]
Nov 03 20:17:15.266306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.266273    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:15.458418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.458340    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b10504a-edeb-4638-86aa-52e54efea88b" path="/var/lib/kubelet/pods/0b10504a-edeb-4638-86aa-52e54efea88b/volumes"
Nov 03 20:17:15.622436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:15.622399    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:16.009181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:16.009150    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:16.623942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:16.623906    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:17:16.624459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:16.623960    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 20:17:16.698854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:16.698817    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:17.410973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:17.410940    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:17.632723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:17.631068    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:17:17.632723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:17.631136    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 20:17:18.115330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:18.115296    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:18.621917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:18.621879    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:17:18.622106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:18.621928    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 20:17:18.834890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:18.834824    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:19.027009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:19.026977    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"]
Nov 03 20:17:19.032152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:19.032125    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbe151ac-m2xl2"]
Nov 03 20:17:19.391774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:19.391685    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:19.556145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:19.556080    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ff6775-eb23-46df-ad3e-e361447f36d3" path="/var/lib/kubelet/pods/18ff6775-eb23-46df-ad3e-e361447f36d3/volumes"
Nov 03 20:17:19.621949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:19.621916    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:17:19.622142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:19.622010    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 20:17:20.111377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:20.111346    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:20.323913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:20.319707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:20.439743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:20.439662    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:20.622799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:20.622766    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:17:20.622976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:20.622819    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 20:17:20.970342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:20.970306    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:21.421707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:21.421677    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:21.633583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:21.633546    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:21.906846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:21.906817    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:21.907385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:21.907357    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:21.909658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:21.909634    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:17:22.386480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:22.386451    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:22.984991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:22.984952    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:23.487757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:23.487727    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"]
Nov 03 20:17:23.487948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:23.487928    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy" containerID="cri-o://d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81" gracePeriod=30
Nov 03 20:17:23.607860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:23.607831    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:23.719066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:23.719028    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"]
Nov 03 20:17:23.719492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:23.719459    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy" containerID="cri-o://913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0" gracePeriod=30
Nov 03 20:17:23.961585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:23.961493    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"]
Nov 03 20:17:23.961729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:23.961698    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy" containerID="cri-o://9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a" gracePeriod=30
Nov 03 20:17:24.381939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:24.381906    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:25.061580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:25.061548    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:25.704294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:25.704257    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:26.377983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:26.377953    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:27.030248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:27.029154    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:27.159839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:27.159804    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:27.763281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:27.763234    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:28.394180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.394142    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:28.867211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.867163    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:17:28.904039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.904001    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"]
Nov 03 20:17:28.905430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.905405    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy"
Nov 03 20:17:28.905539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.905431    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy"
Nov 03 20:17:28.905539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.905468    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy"
Nov 03 20:17:28.905539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.905475    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy"
Nov 03 20:17:28.905766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.905746    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy"
Nov 03 20:17:28.905807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.905785    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" containerName="istio-proxy"
Nov 03 20:17:28.910344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.910299    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.955435    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2c6c50-openshift-default-dockercfg-4phd2\""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956249    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-kube-api-access-b9hkp\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-kube-api-access-b9hkp\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956319    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-istio-token\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-istio-token\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956384    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-data\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956424    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-certs\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956551    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-envoy\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956682    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0901e915-ec8e-45f3-a77c-7087560abf13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0901e915-ec8e-45f3-a77c-7087560abf13-istio-podinfo\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956769    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-socket\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956815    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-credential-socket\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.956894    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0901e915-ec8e-45f3-a77c-7087560abf13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0901e915-ec8e-45f3-a77c-7087560abf13-istiod-ca-cert\") pod \"0901e915-ec8e-45f3-a77c-7087560abf13\" (UID: \"0901e915-ec8e-45f3-a77c-7087560abf13\") "
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.959466    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-certs" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.960173    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-data" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.961252    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-socket" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.961715    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-credential-socket" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.961938    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.961958    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.961971    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.961984    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:28.964540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.962614    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0901e915-ec8e-45f3-a77c-7087560abf13-istiod-ca-cert" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:28.974442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.974404    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-istio-token" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:28.982336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:28.982262    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"]
Nov 03 20:17:29.008687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.008655    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-envoy" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.008840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.008704    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-kube-api-access-b9hkp" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "kube-api-access-b9hkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:29.011401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.011373    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0901e915-ec8e-45f3-a77c-7087560abf13-istio-podinfo" pod "0901e915-ec8e-45f3-a77c-7087560abf13" (UID: "0901e915-ec8e-45f3-a77c-7087560abf13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:29.031009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.030875    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.068832    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.069119    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67151051-1ed2-4a57-a34c-4465e6538648-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.069176    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.069254    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.069293    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.071818    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.071918    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67151051-1ed2-4a57-a34c-4465e6538648-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.072044    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.072154    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qm6v\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-kube-api-access-4qm6v\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.072415    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0901e915-ec8e-45f3-a77c-7087560abf13-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.072437    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0901e915-ec8e-45f3-a77c-7087560abf13-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.072454    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0901e915-ec8e-45f3-a77c-7087560abf13-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.072472    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hkp\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-kube-api-access-b9hkp\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.074218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.072487    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0901e915-ec8e-45f3-a77c-7087560abf13-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.173861    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.173974    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qm6v\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-kube-api-access-4qm6v\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.174041    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.174342    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67151051-1ed2-4a57-a34c-4465e6538648-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.174390    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.174447    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.174479    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.174513    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.174581    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67151051-1ed2-4a57-a34c-4465e6538648-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.175473    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67151051-1ed2-4a57-a34c-4465e6538648-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.175764    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.183219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.176529    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.218985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.216649    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:17:29.225459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.224888    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.225459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.225163    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.226031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.226005    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qm6v\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-kube-api-access-4qm6v\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.228635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.228608    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.237134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.237106    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.237618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.237595    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67151051-1ed2-4a57-a34c-4465e6538648-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.276731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.276606    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"]
Nov 03 20:17:29.277651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.277574    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-token\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-token\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.277780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.277666    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-data\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.277780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.277702    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-credential-socket\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.277780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.277756    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-envoy\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.277953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.277782    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-kube-api-access-ckndv\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-kube-api-access-ckndv\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.277953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.277849    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-podinfo\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.278060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.278010    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-socket\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.278060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.278038    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-certs\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.278154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.278089    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007ac27c-2a36-426d-bc81-a6c70a55454a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007ac27c-2a36-426d-bc81-a6c70a55454a-istiod-ca-cert\") pod \"007ac27c-2a36-426d-bc81-a6c70a55454a\" (UID: \"007ac27c-2a36-426d-bc81-a6c70a55454a\") "
Nov 03 20:17:29.279956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.279904    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007ac27c-2a36-426d-bc81-a6c70a55454a-istiod-ca-cert" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:29.287667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.287637    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-socket" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.289835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.289808    5282 cpu_manager.go:414] "containerMap: removing container" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy"
Nov 03 20:17:29.289974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.289837    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy"
Nov 03 20:17:29.290073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.290057    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerName="istio-proxy"
Nov 03 20:17:29.291467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.291413    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-credential-socket" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.291848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.291830    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-data" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.292030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.291977    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-certs" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.296366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.296340    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-envoy" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.297190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.297133    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-kube-api-access-ckndv" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "kube-api-access-ckndv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:29.299575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.299552    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"]
Nov 03 20:17:29.299769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.299699    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.300327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.300021    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-podinfo" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:29.300327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.300274    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-token" pod "007ac27c-2a36-426d-bc81-a6c70a55454a" (UID: "007ac27c-2a36-426d-bc81-a6c70a55454a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:29.376853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.376823    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:29.380681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.380490    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.380681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.380629    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.380826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.380804    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.380904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.380889    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.380964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.380924    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.381019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.380978    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.381156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381134    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.381239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381227    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5cb8\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-kube-api-access-d5cb8\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.381322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381306    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.381457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381442    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381462    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381478    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007ac27c-2a36-426d-bc81-a6c70a55454a-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381493    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381506    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381520    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381547    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381561    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckndv\" (UniqueName: \"kubernetes.io/projected/007ac27c-2a36-426d-bc81-a6c70a55454a-kube-api-access-ckndv\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.381603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.381576    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007ac27c-2a36-426d-bc81-a6c70a55454a-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.388094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.388073    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:17:29.429314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.429279    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"]
Nov 03 20:17:29.430526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.430500    5282 cpu_manager.go:414] "containerMap: removing container" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy"
Nov 03 20:17:29.430526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.430523    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy"
Nov 03 20:17:29.430728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.430680    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="364a31d7-5518-433e-83ca-5d873b133955" containerName="istio-proxy"
Nov 03 20:17:29.442682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.441949    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.442872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.442854    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"]
Nov 03 20:17:29.446498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.446461    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-46c34c-openshift-default-dockercfg-55bnp\""
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483074    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-certs\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483180    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-data\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483323    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-socket\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483432    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-envoy\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483545    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-credential-socket\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483651    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364a31d7-5518-433e-83ca-5d873b133955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364a31d7-5518-433e-83ca-5d873b133955-istiod-ca-cert\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483766    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-istio-token\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-istio-token\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483801    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/364a31d7-5518-433e-83ca-5d873b133955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364a31d7-5518-433e-83ca-5d873b133955-istio-podinfo\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.483938    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-kube-api-access-hlgj7\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-kube-api-access-hlgj7\") pod \"364a31d7-5518-433e-83ca-5d873b133955\" (UID: \"364a31d7-5518-433e-83ca-5d873b133955\") "
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484432    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484608    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484664    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484695    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484748    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-data" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.484848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484769    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.485740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484867    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.485740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484942    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-certs" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.485740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.484965    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cb8\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-kube-api-access-d5cb8\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.485740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.485063    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.485740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.485232    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-socket" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.485740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.485371    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.485740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.485558    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.486160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.485868    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-credential-socket" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.486160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486076    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364a31d7-5518-433e-83ca-5d873b133955-istiod-ca-cert" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:29.486287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486232    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.486884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486514    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.486884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486674    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.486884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486692    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.486884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486710    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/364a31d7-5518-433e-83ca-5d873b133955-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.486884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486726    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.486884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.486742    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.487610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.487457    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-istio-token" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:29.488216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.488181    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.488895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.488872    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.489060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.489040    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.489769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.489748    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-envoy" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:29.489888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.489845    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-kube-api-access-hlgj7" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "kube-api-access-hlgj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:29.489983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.489895    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/364a31d7-5518-433e-83ca-5d873b133955-istio-podinfo" pod "364a31d7-5518-433e-83ca-5d873b133955" (UID: "364a31d7-5518-433e-83ca-5d873b133955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:29.490991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.490895    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.497138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.496914    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.497138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.497097    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cb8\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-kube-api-access-d5cb8\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.563250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.562359    5282 generic.go:356] "Generic (PLEG): container finished" podID="364a31d7-5518-433e-83ca-5d873b133955" containerID="9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a" exitCode=0
Nov 03 20:17:29.563250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.562418    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"
Nov 03 20:17:29.563250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.562420    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" event={"ID":"364a31d7-5518-433e-83ca-5d873b133955","Type":"ContainerDied","Data":"9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a"}
Nov 03 20:17:29.563250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.562505    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8" event={"ID":"364a31d7-5518-433e-83ca-5d873b133955","Type":"ContainerDied","Data":"1edfe4ece6c8b914b9f44ba662970526827289b40c2c32f945b76bffeed4bdc1"}
Nov 03 20:17:29.563250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.562554    5282 scope.go:122] "RemoveContainer" containerID="9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a"
Nov 03 20:17:29.570801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.570689    5282 generic.go:356] "Generic (PLEG): container finished" podID="007ac27c-2a36-426d-bc81-a6c70a55454a" containerID="913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0" exitCode=0
Nov 03 20:17:29.570938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.570800    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" event={"ID":"007ac27c-2a36-426d-bc81-a6c70a55454a","Type":"ContainerDied","Data":"913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0"}
Nov 03 20:17:29.570938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.570846    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll" event={"ID":"007ac27c-2a36-426d-bc81-a6c70a55454a","Type":"ContainerDied","Data":"2e101b4fc65f4ed2214d49e2b8a36ea970c2ef511acdfb165f9df414f9816650"}
Nov 03 20:17:29.571132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.571057    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"
Nov 03 20:17:29.574208 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.573863    5282 generic.go:356] "Generic (PLEG): container finished" podID="0901e915-ec8e-45f3-a77c-7087560abf13" containerID="d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81" exitCode=0
Nov 03 20:17:29.574208 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.573919    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"
Nov 03 20:17:29.574208 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.574024    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" event={"ID":"0901e915-ec8e-45f3-a77c-7087560abf13","Type":"ContainerDied","Data":"d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81"}
Nov 03 20:17:29.574208 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.574063    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b" event={"ID":"0901e915-ec8e-45f3-a77c-7087560abf13","Type":"ContainerDied","Data":"43f3270c1133af0de1b5c58404549d5ac7344d3379017319978816c1909a1061"}
Nov 03 20:17:29.588040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.588006    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.588480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.588456    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.588743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.588724    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfbjj\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-kube-api-access-gfbjj\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.588991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.588971    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.589128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.589113    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.589422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.589403    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47726d4-a756-464a-9fff-465b9ab355b0-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.589574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.589556    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47726d4-a756-464a-9fff-465b9ab355b0-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.589809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.589787    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.590034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.590017    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.590262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.590246    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/364a31d7-5518-433e-83ca-5d873b133955-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.590368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.590354    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.590476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.590463    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/364a31d7-5518-433e-83ca-5d873b133955-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.590777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.590557    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlgj7\" (UniqueName: \"kubernetes.io/projected/364a31d7-5518-433e-83ca-5d873b133955-kube-api-access-hlgj7\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:29.605745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.605722    5282 scope.go:122] "RemoveContainer" containerID="9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a"
Nov 03 20:17:29.606097 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:17:29.606078    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a\": container with ID starting with 9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a not found: ID does not exist" containerID="9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a"
Nov 03 20:17:29.606155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.606105    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a"} err="failed to get container status \"9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a\": rpc error: code = NotFound desc = could not find container \"9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a\": container with ID starting with 9484512f495bb7d87d5a172a5c30d696d3805e98af2b62936942d6eaa3b30a8a not found: ID does not exist"
Nov 03 20:17:29.606155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.606125    5282 scope.go:122] "RemoveContainer" containerID="913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0"
Nov 03 20:17:29.619990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.619962    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"]
Nov 03 20:17:29.620286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.620258    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:29.623326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.623304    5282 scope.go:122] "RemoveContainer" containerID="913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0"
Nov 03 20:17:29.623698 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:17:29.623667    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0\": container with ID starting with 913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0 not found: ID does not exist" containerID="913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0"
Nov 03 20:17:29.623798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.623696    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0"} err="failed to get container status \"913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0\": rpc error: code = NotFound desc = could not find container \"913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0\": container with ID starting with 913a60fefb2940052bf68d0b01875913b79c1a163771d4e22f8e35127dd271c0 not found: ID does not exist"
Nov 03 20:17:29.623798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.623712    5282 scope.go:122] "RemoveContainer" containerID="d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81"
Nov 03 20:17:29.627467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.627443    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-8cr7b"]
Nov 03 20:17:29.640301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.640274    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"]
Nov 03 20:17:29.644591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.644569    5282 scope.go:122] "RemoveContainer" containerID="ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0"
Nov 03 20:17:29.644885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.644864    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-6gxll"]
Nov 03 20:17:29.676386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.676349    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"]
Nov 03 20:17:29.683565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.683531    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-tl2q8"]
Nov 03 20:17:29.692354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692241    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47726d4-a756-464a-9fff-465b9ab355b0-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692285    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47726d4-a756-464a-9fff-465b9ab355b0-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692400    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692438    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692471    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692556    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692598    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfbjj\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-kube-api-access-gfbjj\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692695    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.692724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.692721    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.693167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.693141    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.693551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.693526    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.693637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.693537    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.693765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.693749    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.696442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.696416    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.696726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.696700    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47726d4-a756-464a-9fff-465b9ab355b0-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.705712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.705432    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47726d4-a756-464a-9fff-465b9ab355b0-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.705712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.705444    5282 scope.go:122] "RemoveContainer" containerID="d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81"
Nov 03 20:17:29.705712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.705591    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.706987 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:17:29.706910    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81\": container with ID starting with d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81 not found: ID does not exist" containerID="d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81"
Nov 03 20:17:29.706987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.706942    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81"} err="failed to get container status \"d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81\": rpc error: code = NotFound desc = could not find container \"d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81\": container with ID starting with d984af0f8dc469927ff00802bc84cd2007900c747794efd9851a7bcf96059b81 not found: ID does not exist"
Nov 03 20:17:29.706987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.706968    5282 scope.go:122] "RemoveContainer" containerID="ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0"
Nov 03 20:17:29.708608 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:17:29.707719    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0\": container with ID starting with ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0 not found: ID does not exist" containerID="ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0"
Nov 03 20:17:29.708608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.707768    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0"} err="failed to get container status \"ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0\": rpc error: code = NotFound desc = could not find container \"ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0\": container with ID starting with ab80de0f05c72ddf96847f50fa763d0e9daa21d56afb204445177fc86d2197b0 not found: ID does not exist"
Nov 03 20:17:29.708774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.708730    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfbjj\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-kube-api-access-gfbjj\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.721399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.718166    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"]
Nov 03 20:17:29.749164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.749112    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:29.749295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.749234    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:29.749295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.749287    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:29.794875 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.794722    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:29.796535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.796512    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:29.965135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.965083    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"]
Nov 03 20:17:29.975953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.975900    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:29.976791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.976728    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:29.976974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:29.976954    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:30.243368 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:17:30.243336    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47726d4_a756_464a_9fff_465b9ab355b0.slice/crio-c1cb45b8c8d0f89cb9fe1c6e622d2eee055e77c4a6da7eadc29e987f757c7d71 WatchSource:0}: Error finding container c1cb45b8c8d0f89cb9fe1c6e622d2eee055e77c4a6da7eadc29e987f757c7d71: Status 404 returned error can't find the container with id c1cb45b8c8d0f89cb9fe1c6e622d2eee055e77c4a6da7eadc29e987f757c7d71
Nov 03 20:17:30.245687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.245653    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"]
Nov 03 20:17:30.248823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.248786    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:30.248993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.248974    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:30.249122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.249106    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:30.418415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.418380    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:30.614509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.614421    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" event={"ID":"55c2b3db-5451-4b5a-8313-9fc3fb58db6b","Type":"ContainerStarted","Data":"cda6307b4bcf70d1581e8da93a7cd4f33cde5449f55c47814dd61a7fe8f69f82"}
Nov 03 20:17:30.620227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.619709    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" event={"ID":"55c2b3db-5451-4b5a-8313-9fc3fb58db6b","Type":"ContainerStarted","Data":"38a19c0cd8e00feb804c1251219fb869e2ac3bbcfe6005eac88aec63b98f36d3"}
Nov 03 20:17:30.620757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.620584    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:30.621535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.621413    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" event={"ID":"67151051-1ed2-4a57-a34c-4465e6538648","Type":"ContainerStarted","Data":"347d98c8e88fe80a339fed93aa1e89b09546027c3a9c6140bfc7c35d827d6a5a"}
Nov 03 20:17:30.621535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.621452    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" event={"ID":"67151051-1ed2-4a57-a34c-4465e6538648","Type":"ContainerStarted","Data":"9aa0524e1c6d47ff231697444c4e99ff52993d5afe402f62ae09ca8212573e18"}
Nov 03 20:17:30.623661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.623639    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" event={"ID":"f47726d4-a756-464a-9fff-465b9ab355b0","Type":"ContainerStarted","Data":"9249a1cfd31aa77756189e5fe6182125ac13a461020fb11578ed6bb72ffc2a5b"}
Nov 03 20:17:30.624119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.624099    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" event={"ID":"f47726d4-a756-464a-9fff-465b9ab355b0","Type":"ContainerStarted","Data":"c1cb45b8c8d0f89cb9fe1c6e622d2eee055e77c4a6da7eadc29e987f757c7d71"}
Nov 03 20:17:30.642438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.642368    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podStartSLOduration=69.642349555 podStartE2EDuration="1m9.642349555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:30.638988829 +0000 UTC m=+2243.308190291" watchObservedRunningTime="2026-11-03 20:17:30.642349555 +0000 UTC m=+2243.311551024"
Nov 03 20:17:30.658671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.658606    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podStartSLOduration=69.6585877 podStartE2EDuration="1m9.6585877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:30.656440983 +0000 UTC m=+2243.325642467" watchObservedRunningTime="2026-11-03 20:17:30.6585877 +0000 UTC m=+2243.327789169"
Nov 03 20:17:30.680505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.680422    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podStartSLOduration=116.680399872 podStartE2EDuration="1m56.680399872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:30.677582251 +0000 UTC m=+2243.346783721" watchObservedRunningTime="2026-11-03 20:17:30.680399872 +0000 UTC m=+2243.349601345"
Nov 03 20:17:30.798457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:30.798357    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:31.096670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.096620    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:31.410299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.408614    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:31.419345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.418428    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:17:31.419345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.418493    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:17:31.458966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.458927    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007ac27c-2a36-426d-bc81-a6c70a55454a" path="/var/lib/kubelet/pods/007ac27c-2a36-426d-bc81-a6c70a55454a/volumes"
Nov 03 20:17:31.459665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.459641    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0901e915-ec8e-45f3-a77c-7087560abf13" path="/var/lib/kubelet/pods/0901e915-ec8e-45f3-a77c-7087560abf13/volumes"
Nov 03 20:17:31.460257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.460237    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364a31d7-5518-433e-83ca-5d873b133955" path="/var/lib/kubelet/pods/364a31d7-5518-433e-83ca-5d873b133955/volumes"
Nov 03 20:17:31.647265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.637834    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:17:31.647265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.637906    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:17:31.661111 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:17:31.661024    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:17:31.748186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.748150    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:31.820632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:31.820594    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:32.396222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:32.395294    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:17:32.396222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:32.395361    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:17:32.499221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:32.497663    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:32.644221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:32.629859    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:17:32.644221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:32.629941    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:17:32.820037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:32.820007    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:32.835333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:32.835302    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:17:33.033936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:33.033898    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:33.162124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:33.162042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:33.390222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:33.384588    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:17:33.390222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:33.384690    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:17:33.406251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:33.402965    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:33.626220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:33.625888    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:17:33.626220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:33.625955    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:17:34.199225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:34.198163    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:34.389246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:34.388509    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:17:34.389246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:34.388568    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:17:34.449222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:34.415666    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:34.691228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:34.679551    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:17:34.691228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:34.679605    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:17:35.021855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:35.021819    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:35.400433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:35.400337    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:17:35.401045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:35.401013    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:17:35.625188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:35.624630    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:17:35.625188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:35.624691    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:17:35.700281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:35.699105    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:36.379401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:36.379354    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:36.384704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:36.384670    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:17:36.384941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:36.384911    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 20:17:36.621441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:36.621286    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:17:36.621441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:36.621357    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 20:17:37.007923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.007860    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:37.382533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.382459    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:37.383020 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.382999    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:37.383755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.383733    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:17:37.626747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.626720    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:37.627218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.627186    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:37.627978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.627956    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:17:37.822609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:37.809607    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:38.380970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:38.380922    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:39.001848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:39.001807    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:39.692449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:39.692415    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:40.301555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:40.301510    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:40.948717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:40.948685    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:41.689920 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:41.689888    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:42.389335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:42.389283    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:42.995342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:42.995314    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:43.687015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:43.686985    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:44.382384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:44.382348    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:44.989457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:44.989386    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:45.824894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:45.824859    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:46.395145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:46.395113    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.017987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:47.017952    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:47.710998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:47.710961    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:48.410918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:48.410868    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:49.046639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.046603    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"]
Nov 03 20:17:49.046861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.046832    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy" containerID="cri-o://097866ceb1f054b3bbc8921db1ee70982b95e515f27fca67463f0b326e1f3cce" gracePeriod=30
Nov 03 20:17:49.052430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.052407    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"]
Nov 03 20:17:49.052622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.052603    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy" containerID="cri-o://c59e758ff4635fe43931403b6cf45960cca9b1c7b87476ed327666c3868e9bb8" gracePeriod=30
Nov 03 20:17:49.065583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.065502    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:49.178962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.178933    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:49.324949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.324858    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:49.934886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:49.934851    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:50.805704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:50.805674    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:51.497985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:51.497948    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:52.108113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:52.108079    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:53.048856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:53.048817    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:53.665331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:53.665283    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:54.318039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.317956    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:54.380747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.380453    5282 generic.go:356] "Generic (PLEG): container finished" podID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerID="097866ceb1f054b3bbc8921db1ee70982b95e515f27fca67463f0b326e1f3cce" exitCode=0
Nov 03 20:17:54.380914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.380532    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" event={"ID":"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce","Type":"ContainerDied","Data":"097866ceb1f054b3bbc8921db1ee70982b95e515f27fca67463f0b326e1f3cce"}
Nov 03 20:17:54.380914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.380842    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d" event={"ID":"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce","Type":"ContainerDied","Data":"94b929caa4e5320f0632bd8787dddef46b5b411509d16170537d881fc14ca571"}
Nov 03 20:17:54.380914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.380863    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94b929caa4e5320f0632bd8787dddef46b5b411509d16170537d881fc14ca571"
Nov 03 20:17:54.383906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.383878    5282 generic.go:356] "Generic (PLEG): container finished" podID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerID="c59e758ff4635fe43931403b6cf45960cca9b1c7b87476ed327666c3868e9bb8" exitCode=0
Nov 03 20:17:54.384047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.384034    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" event={"ID":"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c","Type":"ContainerDied","Data":"c59e758ff4635fe43931403b6cf45960cca9b1c7b87476ed327666c3868e9bb8"}
Nov 03 20:17:54.385466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.385443    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:17:54.432256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.432215    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"]
Nov 03 20:17:54.434147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.434115    5282 cpu_manager.go:414] "containerMap: removing container" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy"
Nov 03 20:17:54.434351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.434335    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy"
Nov 03 20:17:54.434844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.434822    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" containerName="istio-proxy"
Nov 03 20:17:54.461012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.460977    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-data\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.461342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.461312    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-token\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-token\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.461525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.461508    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-certs\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.461743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.461726    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-envoy\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.461931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.461914    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-kube-api-access-v4n89\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-kube-api-access-v4n89\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.462242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.462183    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-podinfo\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.462482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.462465    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istiod-ca-cert\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.462613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.462600    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-credential-socket\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.462889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.462876    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-socket\") pod \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\" (UID: \"df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce\") "
Nov 03 20:17:54.464171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.464149    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-socket" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.464533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.464516    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-data" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.470181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.465479    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.470181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.467638    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-certs" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.470181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.469105    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-credential-socket" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.470181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.470113    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istiod-ca-cert" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:54.532360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.518249    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:17:54.535075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.533970    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-token" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:54.552220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.545611    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-podinfo" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:54.552220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.551598    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-envoy" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.556249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.554854    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"]
Nov 03 20:17:54.556249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.555357    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-kube-api-access-v4n89" pod "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" (UID: "df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce"). InnerVolumeSpecName "kube-api-access-v4n89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:54.632032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.631996    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-podinfo\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.632357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.632324    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-envoy\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.632604 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.632582    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-credential-socket\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.637262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.636332    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-socket\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.637262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.636487    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-certs\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.637262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.636738    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istiod-ca-cert\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.637262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.636796    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-kube-api-access-bj9xz\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-kube-api-access-bj9xz\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.637262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.636838    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-token\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-token\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.637262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.636929    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-data\") pod \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\" (UID: \"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c\") "
Nov 03 20:17:54.639330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.632994    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-credential-socket" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.639591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.639563    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-certs" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.644245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.643402    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-token" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:54.645299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.644884    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-data" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.646028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.645963    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-socket" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.646134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.646110    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istiod-ca-cert" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:54.646595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.646566    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.653392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.653367    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-podinfo" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.672808    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.673015    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.673052    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f48c89-4498-48ff-8c32-4a2c796151f1-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.673129    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.673372    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.675940    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676047    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676227    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmchq\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-kube-api-access-wmchq\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676602    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676640    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676659    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676679    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676706    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676722    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676735    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676753    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676768    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4n89\" (UniqueName: \"kubernetes.io/projected/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-kube-api-access-v4n89\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676785    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676801    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676821    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676837    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676853    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676869    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.672860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.676888    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.687557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.687509    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-envoy" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:17:54.689260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.687883    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"]
Nov 03 20:17:54.689631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.689601    5282 cpu_manager.go:414] "containerMap: removing container" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy"
Nov 03 20:17:54.689756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.689744    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy"
Nov 03 20:17:54.690161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.690147    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" containerName="istio-proxy"
Nov 03 20:17:54.690467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.690437    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-kube-api-access-bj9xz" pod "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" (UID: "bfea1a5a-61d8-462d-9095-a75fb7d5eb1c"). InnerVolumeSpecName "kube-api-access-bj9xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:54.695703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.695682    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"]
Nov 03 20:17:54.695820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.695800    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:54.699679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.699656    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:17:54.700446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.700424    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-46c34c\""
Nov 03 20:17:54.721626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.721557    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-46c34c-dockercfg-bz8h2\""
Nov 03 20:17:54.778377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778339    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") " pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:54.778532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778393    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.778600 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778523    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmchq\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-kube-api-access-wmchq\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.778670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778649    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.778832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778816    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.778920 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778904    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.778969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778932    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f48c89-4498-48ff-8c32-4a2c796151f1-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778970    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.778998    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-data\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779009    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8lj\" (UniqueName: \"kubernetes.io/projected/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-kube-api-access-dr8lj\") pod \"payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") " pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:54.779164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779143    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779180    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779192    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779280    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-credential-socket\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779398    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.779471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779409    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-certs\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.779471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779426    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj9xz\" (UniqueName: \"kubernetes.io/projected/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c-kube-api-access-bj9xz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:54.779628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.779607    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f48c89-4498-48ff-8c32-4a2c796151f1-istiod-ca-cert\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.781421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.781397    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-envoy\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.781653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.781633    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-podinfo\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.786494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.786469    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-token\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.786695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.786674    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmchq\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-kube-api-access-wmchq\") pod \"e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") " pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:54.881328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.881238    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8lj\" (UniqueName: \"kubernetes.io/projected/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-kube-api-access-dr8lj\") pod \"payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") " pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:54.881494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.881340    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") " pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:54.882067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.882046    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") " pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:54.889153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.889126    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8lj\" (UniqueName: \"kubernetes.io/projected/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-kube-api-access-dr8lj\") pod \"payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") " pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:54.945095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:54.945052    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:55.035852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.035632    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:55.108930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.108905    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:55.301227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.300835    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:55.301227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.300939    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:55.301227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.300976    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:17:55.312262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.312216    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"]
Nov 03 20:17:55.396684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.396655    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" event={"ID":"33f48c89-4498-48ff-8c32-4a2c796151f1","Type":"ContainerStarted","Data":"a348531e6a7be7896843c4f4d2f4d3a6ecfa806645863d3bd3d79fe567c7bc80"}
Nov 03 20:17:55.398297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.398274    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"
Nov 03 20:17:55.398414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.398293    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j" event={"ID":"bfea1a5a-61d8-462d-9095-a75fb7d5eb1c","Type":"ContainerDied","Data":"9d92d32b5a8c385badc258caa811cd511455d818bba50a4d9111423b4ddef8f5"}
Nov 03 20:17:55.398414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.398351    5282 scope.go:122] "RemoveContainer" containerID="c59e758ff4635fe43931403b6cf45960cca9b1c7b87476ed327666c3868e9bb8"
Nov 03 20:17:55.398601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.398577    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"
Nov 03 20:17:55.432279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.432244    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"]
Nov 03 20:17:55.436877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.436844    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-gfk8j"]
Nov 03 20:17:55.445972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.445937    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"]
Nov 03 20:17:55.461489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.461454    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfea1a5a-61d8-462d-9095-a75fb7d5eb1c" path="/var/lib/kubelet/pods/bfea1a5a-61d8-462d-9095-a75fb7d5eb1c/volumes"
Nov 03 20:17:55.461977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.461958    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-nws6d"]
Nov 03 20:17:55.527778 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.527747    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"]
Nov 03 20:17:55.793351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:55.788690    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:56.500096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:56.500055    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" event={"ID":"33f48c89-4498-48ff-8c32-4a2c796151f1","Type":"ContainerStarted","Data":"14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1"}
Nov 03 20:17:56.507531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:56.507499    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" event={"ID":"28ca8d9b-bda8-46c9-bb3e-69aa77e06445","Type":"ContainerStarted","Data":"31df442ff7e6f3b98ff956bee23e7596443ca2ffba45e157784bfd876e8290ad"}
Nov 03 20:17:56.507780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:56.507759    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" event={"ID":"28ca8d9b-bda8-46c9-bb3e-69aa77e06445","Type":"ContainerStarted","Data":"526f90cace1d3dbb6c2995d16336e68c9379d692267a89778c2d6843ccb62125"}
Nov 03 20:17:56.508669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:56.508647    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:17:56.565221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:56.561310    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podStartSLOduration=140.561285642 podStartE2EDuration="2m20.561285642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:56.539033403 +0000 UTC m=+2269.208234934" watchObservedRunningTime="2026-11-03 20:17:56.561285642 +0000 UTC m=+2269.230487112"
Nov 03 20:17:56.593223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:56.587698    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" podStartSLOduration=22.587675162 podStartE2EDuration="22.587675162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:56.576178806 +0000 UTC m=+2269.245380277" watchObservedRunningTime="2026-11-03 20:17:56.587675162 +0000 UTC m=+2269.256876633"
Nov 03 20:17:56.857269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:56.856749    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:57.007147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.004601    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:57.058482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.055129    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:17:57.083247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.058788    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:17:57.083247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.058839    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:17:57.283279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.282379    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:57.344602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.340321    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:57.458634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.458591    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce" path="/var/lib/kubelet/pods/df4a2e6f-3a4c-4b2a-ba5b-4fafdf736cce/volumes"
Nov 03 20:17:57.604534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.604436    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:57.945855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.945773    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:17:57.945855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:57.945824    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:17:58.319219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:58.318774    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:58.945603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:58.945570    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:17:58.946115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:58.945621    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:17:59.162538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:59.162508    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:59.900430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:59.900399    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:17:59.951641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:59.951601    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:17:59.952290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:17:59.952249    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:18:00.773391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:00.773352    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:00.945897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:00.945857    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:18:00.946099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:00.945918    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:18:01.208085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:01.208012    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:01.626667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:01.626629    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:01.636894 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:18:01.636858    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:18:01.946040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:01.945926    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:18:01.946040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:01.946002    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 20:18:02.450287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:02.450252    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:02.950678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:02.950647    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:18:02.951128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:02.951105    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:18:02.951837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:02.951818    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:18:03.268417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:03.268384    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:04.103812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:04.103778    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:04.897569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:04.897516    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:05.838429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:05.838396    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:06.706792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:06.706757    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:07.484285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:07.484236    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:08.162439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:08.162388    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:08.527992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:08.527964    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:18:08.909516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:08.909424    5282 scope.go:122] "RemoveContainer" containerID="82d9b96ab0e52396b51584df790a42d767375b17a8fc6f044855690ae3196748"
Nov 03 20:18:08.921122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:08.921101    5282 scope.go:122] "RemoveContainer" containerID="097866ceb1f054b3bbc8921db1ee70982b95e515f27fca67463f0b326e1f3cce"
Nov 03 20:18:08.936485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:08.936459    5282 scope.go:122] "RemoveContainer" containerID="afa7739934c389042c9171d889ec978c83d00c9b59de3bd7e9995f004a7801aa"
Nov 03 20:18:08.947765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:08.947735    5282 scope.go:122] "RemoveContainer" containerID="14daa48c7d1d6af450d6433427be1c44d4272368d233f1d8bfa1b39e5ff03948"
Nov 03 20:18:08.987726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:08.987700    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:10.077535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:10.077507    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:10.813522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:10.813491    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:10.926590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:10.922892    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:11.610946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:11.610919    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:12.396916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:12.396878    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:13.072900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:13.072870    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:13.672923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:13.672891    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:13.812660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:13.812624    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:14.128711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:14.128673    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:14.320936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:14.320906    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:14.590118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:14.590087    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:15.063218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:15.058138    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:15.386526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:15.386443    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:15.521609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:15.521321    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:16.192308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:16.192269    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:16.941222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:16.940827    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:17.593480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:17.593449    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:18.206047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:18.206011    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:18.821105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:18.821070    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:18.973763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:18.973727    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:19.087331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:19.087233    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:19.524284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:19.524238    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:20.196705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:20.196667    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:20.822392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:20.822351    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:21.530758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.526741    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:21.587625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.587593    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"]
Nov 03 20:18:21.594544 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.594519    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.602358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.600446    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:21.602358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.600527    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:18:21.602358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.602266    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"]
Nov 03 20:18:21.624167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.624138    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.624315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.624177    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfww6\" (UniqueName: \"kubernetes.io/projected/42d0a919-e7ef-480f-b9e7-523f5970b16d-kube-api-access-cfww6\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.624498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.624470    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.686293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.686252    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:21.797598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.797514    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.798268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.798238    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.798445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.798430    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfww6\" (UniqueName: \"kubernetes.io/projected/42d0a919-e7ef-480f-b9e7-523f5970b16d-kube-api-access-cfww6\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.806815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.806773    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.826044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.826001    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.828391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.828332    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfww6\" (UniqueName: \"kubernetes.io/projected/42d0a919-e7ef-480f-b9e7-523f5970b16d-kube-api-access-cfww6\") pod \"maas-api-revoke-sub-e92a75f2-vb6cs\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.916637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.916609    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:21.949730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.949699    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"]
Nov 03 20:18:21.960420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.960395    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:21.966390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:21.966362    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"]
Nov 03 20:18:22.105296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.105185    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.105481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.105387    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-kube-api-access-qfnzr\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.105545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.105521    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.207295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.207256    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.207468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.207351    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-kube-api-access-qfnzr\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.207468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.207436    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.208119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.208092    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.209845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.209818    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.215559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.215535    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-kube-api-access-qfnzr\") pod \"maas-api-revoke-sub-4590a217-9f2jb\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.233244 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:18:22.233209    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d0a919_e7ef_480f_b9e7_523f5970b16d.slice/crio-7e6c901e954d66fe1df2bf167078a155015e8f9425efe8c2280a3ffd555a3479 WatchSource:0}: Error finding container 7e6c901e954d66fe1df2bf167078a155015e8f9425efe8c2280a3ffd555a3479: Status 404 returned error can't find the container with id 7e6c901e954d66fe1df2bf167078a155015e8f9425efe8c2280a3ffd555a3479
Nov 03 20:18:22.235253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.235213    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"]
Nov 03 20:18:22.285270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.285243    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:22.409248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.409163    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:22.618639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:22.618436    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"]
Nov 03 20:18:22.676616 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:18:22.676569    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fdfebc_775b_49c1_b020_d0cb71b7e4dd.slice/crio-91b7d12f4ce1c2d52edcf3d5c3592d6c4be49920e0817227b8f656551d66043d WatchSource:0}: Error finding container 91b7d12f4ce1c2d52edcf3d5c3592d6c4be49920e0817227b8f656551d66043d: Status 404 returned error can't find the container with id 91b7d12f4ce1c2d52edcf3d5c3592d6c4be49920e0817227b8f656551d66043d
Nov 03 20:18:23.020910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.020873    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs" event={"ID":"42d0a919-e7ef-480f-b9e7-523f5970b16d","Type":"ContainerStarted","Data":"a5b4d7254dfffc61714c397c44bc0abb23ecc1ff4e89ec8ce84a65c5743e3586"}
Nov 03 20:18:23.021382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.020923    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs" event={"ID":"42d0a919-e7ef-480f-b9e7-523f5970b16d","Type":"ContainerStarted","Data":"7e6c901e954d66fe1df2bf167078a155015e8f9425efe8c2280a3ffd555a3479"}
Nov 03 20:18:23.022245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.022219    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb" event={"ID":"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd","Type":"ContainerStarted","Data":"91b7d12f4ce1c2d52edcf3d5c3592d6c4be49920e0817227b8f656551d66043d"}
Nov 03 20:18:23.068771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.067700    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:23.094219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.087007    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs" podStartSLOduration=2.086987979 podStartE2EDuration="2.086987979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:23.068265002 +0000 UTC m=+2295.737466498" watchObservedRunningTime="2026-11-03 20:18:23.086987979 +0000 UTC m=+2295.756189472"
Nov 03 20:18:23.711161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.711080    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:23.879322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.878654    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"]
Nov 03 20:18:23.879322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:23.878999    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy" containerID="cri-o://c7f738898274e30450c09fc31dab84a61f4d7049166afc8f4beeca2b0951fea4" gracePeriod=30
Nov 03 20:18:24.032444 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:24.032404    5282 generic.go:356] "Generic (PLEG): container finished" podID="42d0a919-e7ef-480f-b9e7-523f5970b16d" containerID="a5b4d7254dfffc61714c397c44bc0abb23ecc1ff4e89ec8ce84a65c5743e3586" exitCode=0
Nov 03 20:18:24.032870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:24.032497    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs" event={"ID":"42d0a919-e7ef-480f-b9e7-523f5970b16d","Type":"ContainerDied","Data":"a5b4d7254dfffc61714c397c44bc0abb23ecc1ff4e89ec8ce84a65c5743e3586"}
Nov 03 20:18:24.034431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:24.034403    5282 generic.go:356] "Generic (PLEG): container finished" podID="d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" containerID="a79fce7003093f237e61b81360d8355548ddc52e02163d91954fbf46591749f9" exitCode=0
Nov 03 20:18:24.034706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:24.034691    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb" event={"ID":"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd","Type":"ContainerDied","Data":"a79fce7003093f237e61b81360d8355548ddc52e02163d91954fbf46591749f9"}
Nov 03 20:18:24.646955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:24.645687    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:25.182826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.155731    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:25.414788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.414756    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:25.418820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.418794    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:25.491325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.491298    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-openshift-ca\") pod \"42d0a919-e7ef-480f-b9e7-523f5970b16d\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") "
Nov 03 20:18:25.491484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.491387    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-xks-ca\") pod \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") "
Nov 03 20:18:25.491484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.491460    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-kube-api-access-qfnzr\") pod \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") "
Nov 03 20:18:25.491617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.491557    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d0a919-e7ef-480f-b9e7-523f5970b16d-kube-api-access-cfww6\" (UniqueName: \"kubernetes.io/projected/42d0a919-e7ef-480f-b9e7-523f5970b16d-kube-api-access-cfww6\") pod \"42d0a919-e7ef-480f-b9e7-523f5970b16d\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") "
Nov 03 20:18:25.491677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.491638    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-openshift-ca\") pod \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\" (UID: \"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd\") "
Nov 03 20:18:25.491731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.491670    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-openshift-ca" pod "42d0a919-e7ef-480f-b9e7-523f5970b16d" (UID: "42d0a919-e7ef-480f-b9e7-523f5970b16d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:25.491783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.491732    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-xks-ca\") pod \"42d0a919-e7ef-480f-b9e7-523f5970b16d\" (UID: \"42d0a919-e7ef-480f-b9e7-523f5970b16d\") "
Nov 03 20:18:25.492185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.492159    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-openshift-ca" pod "d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" (UID: "d5fdfebc-775b-49c1-b020-d0cb71b7e4dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:25.492932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.492883    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:25.492932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.492909    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:25.494523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.494498    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-kube-api-access-qfnzr" pod "d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" (UID: "d5fdfebc-775b-49c1-b020-d0cb71b7e4dd"). InnerVolumeSpecName "kube-api-access-qfnzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:25.494636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.494572    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-xks-ca" pod "d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" (UID: "d5fdfebc-775b-49c1-b020-d0cb71b7e4dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:25.494922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.494896    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-xks-ca" pod "42d0a919-e7ef-480f-b9e7-523f5970b16d" (UID: "42d0a919-e7ef-480f-b9e7-523f5970b16d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:25.496432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.496411    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d0a919-e7ef-480f-b9e7-523f5970b16d-kube-api-access-cfww6" pod "42d0a919-e7ef-480f-b9e7-523f5970b16d" (UID: "42d0a919-e7ef-480f-b9e7-523f5970b16d"). InnerVolumeSpecName "kube-api-access-cfww6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:25.594628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.594597    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:25.594628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.594627    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfnzr\" (UniqueName: \"kubernetes.io/projected/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd-kube-api-access-qfnzr\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:25.594798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.594637    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfww6\" (UniqueName: \"kubernetes.io/projected/42d0a919-e7ef-480f-b9e7-523f5970b16d-kube-api-access-cfww6\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:25.594798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.594647    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d0a919-e7ef-480f-b9e7-523f5970b16d-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:25.782232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:25.782182    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:26.058370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.058288    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"
Nov 03 20:18:26.058516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.058295    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs" event={"ID":"42d0a919-e7ef-480f-b9e7-523f5970b16d","Type":"ContainerDied","Data":"7e6c901e954d66fe1df2bf167078a155015e8f9425efe8c2280a3ffd555a3479"}
Nov 03 20:18:26.058516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.058415    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e6c901e954d66fe1df2bf167078a155015e8f9425efe8c2280a3ffd555a3479"
Nov 03 20:18:26.059819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.059797    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"
Nov 03 20:18:26.059941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.059837    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb" event={"ID":"d5fdfebc-775b-49c1-b020-d0cb71b7e4dd","Type":"ContainerDied","Data":"91b7d12f4ce1c2d52edcf3d5c3592d6c4be49920e0817227b8f656551d66043d"}
Nov 03 20:18:26.059941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.059883    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91b7d12f4ce1c2d52edcf3d5c3592d6c4be49920e0817227b8f656551d66043d"
Nov 03 20:18:26.395971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.395880    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:26.972379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:26.972347    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:27.604259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:27.604225    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:28.305647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:28.305616    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:28.957241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:28.956869    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:29.124724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.124681    5282 generic.go:356] "Generic (PLEG): container finished" podID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerID="c7f738898274e30450c09fc31dab84a61f4d7049166afc8f4beeca2b0951fea4" exitCode=0
Nov 03 20:18:29.124889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.124734    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" event={"ID":"32c977e2-140f-4591-9f8c-78361d36ef0f","Type":"ContainerDied","Data":"c7f738898274e30450c09fc31dab84a61f4d7049166afc8f4beeca2b0951fea4"}
Nov 03 20:18:29.280284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.280253    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:18:29.311758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.311726    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"]
Nov 03 20:18:29.312653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312635    5282 cpu_manager.go:414] "containerMap: removing container" podUID="42d0a919-e7ef-480f-b9e7-523f5970b16d" containerName="revoke-keys"
Nov 03 20:18:29.312709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312654    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d0a919-e7ef-480f-b9e7-523f5970b16d" containerName="revoke-keys"
Nov 03 20:18:29.312709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312683    5282 cpu_manager.go:414] "containerMap: removing container" podUID="d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" containerName="revoke-keys"
Nov 03 20:18:29.312709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312689    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" containerName="revoke-keys"
Nov 03 20:18:29.312709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312696    5282 cpu_manager.go:414] "containerMap: removing container" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy"
Nov 03 20:18:29.312709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312702    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy"
Nov 03 20:18:29.312869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312835    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d0a919-e7ef-480f-b9e7-523f5970b16d" containerName="revoke-keys"
Nov 03 20:18:29.312869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312848    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" containerName="revoke-keys"
Nov 03 20:18:29.312869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.312854    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" containerName="istio-proxy"
Nov 03 20:18:29.316963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.316938    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.322107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.321911    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a7343-openshift-default-dockercfg-qgfmb\""
Nov 03 20:18:29.339395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.339364    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"]
Nov 03 20:18:29.400529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.400497    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-podinfo\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.400708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.400541    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-socket\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.400708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.400593    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-credential-socket\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.400844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.400817    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-socket" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:29.401488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.400939    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-credential-socket" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:29.401488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401032    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32c977e2-140f-4591-9f8c-78361d36ef0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c977e2-140f-4591-9f8c-78361d36ef0f-istiod-ca-cert\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.401488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401069    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-envoy\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.401488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401149    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-certs\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.401488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401268    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-data\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.401488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401369    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c977e2-140f-4591-9f8c-78361d36ef0f-istiod-ca-cert" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:29.401488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401476    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-token\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-token\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.401935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401531    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-kube-api-access-skn47\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-kube-api-access-skn47\") pod \"32c977e2-140f-4591-9f8c-78361d36ef0f\" (UID: \"32c977e2-140f-4591-9f8c-78361d36ef0f\") "
Nov 03 20:18:29.401935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401598    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-certs" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:29.401935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401743    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-data" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:29.401935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.401906    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6ft\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-kube-api-access-vt6ft\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402067    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-data\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402225    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402449    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-certs\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402490    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-envoy\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402535    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de77257b-93d9-472b-a007-a6d199acf1e2-istio-podinfo\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402573    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-istio-token\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402632    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-credential-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402673    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de77257b-93d9-472b-a007-a6d199acf1e2-istiod-ca-cert\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.402831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402816    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.402906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402837    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.402906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402852    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c977e2-140f-4591-9f8c-78361d36ef0f-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.402906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402864    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.402906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.402879    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.403772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.403751    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-podinfo" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:18:29.404028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.404008    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-token" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:29.404116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.404076    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-envoy" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:29.404240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.404182    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-kube-api-access-skn47" pod "32c977e2-140f-4591-9f8c-78361d36ef0f" (UID: "32c977e2-140f-4591-9f8c-78361d36ef0f"). InnerVolumeSpecName "kube-api-access-skn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:29.504881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.504809    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-data\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.504969    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505155    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-certs\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505190    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-envoy\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505224    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-data\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505245    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de77257b-93d9-472b-a007-a6d199acf1e2-istio-podinfo\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505298    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-istio-token\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505363    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-credential-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505406    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de77257b-93d9-472b-a007-a6d199acf1e2-istiod-ca-cert\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505302    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505500    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6ft\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-kube-api-access-vt6ft\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505681    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.505812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505699    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skn47\" (UniqueName: \"kubernetes.io/projected/32c977e2-140f-4591-9f8c-78361d36ef0f-kube-api-access-skn47\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.505812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505716    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.505812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505729    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c977e2-140f-4591-9f8c-78361d36ef0f-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.505812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505769    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-credential-socket\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.505992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.505844    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-certs\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.506128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.506105    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de77257b-93d9-472b-a007-a6d199acf1e2-istiod-ca-cert\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.507564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.507543    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-envoy\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.507823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.507800    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de77257b-93d9-472b-a007-a6d199acf1e2-istio-podinfo\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.513607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.513581    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-istio-token\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.513736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.513718    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6ft\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-kube-api-access-vt6ft\") pod \"e2e-inf-b-0a7343-openshift-default-74585db445-lplwl\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") " pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:29.608807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.608772    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:29.668834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:29.668383    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:30.044056 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:18:30.044027    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde77257b_93d9_472b_a007_a6d199acf1e2.slice/crio-795106f6d51744ee6e1fee8954ea0404348374fa6b11ff5a229bdfbcf9f13607 WatchSource:0}: Error finding container 795106f6d51744ee6e1fee8954ea0404348374fa6b11ff5a229bdfbcf9f13607: Status 404 returned error can't find the container with id 795106f6d51744ee6e1fee8954ea0404348374fa6b11ff5a229bdfbcf9f13607
Nov 03 20:18:30.045403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.045374    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"]
Nov 03 20:18:30.047526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.047493    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:18:30.047602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.047554    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:18:30.047602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.047585    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:18:30.135863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.135836    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"
Nov 03 20:18:30.136009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.135840    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p" event={"ID":"32c977e2-140f-4591-9f8c-78361d36ef0f","Type":"ContainerDied","Data":"1814a32c1750bea109628c2ba48dfa3c4b487d082d738058db83974efe651035"}
Nov 03 20:18:30.136009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.135978    5282 scope.go:122] "RemoveContainer" containerID="c7f738898274e30450c09fc31dab84a61f4d7049166afc8f4beeca2b0951fea4"
Nov 03 20:18:30.137486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.137458    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" event={"ID":"de77257b-93d9-472b-a007-a6d199acf1e2","Type":"ContainerStarted","Data":"795106f6d51744ee6e1fee8954ea0404348374fa6b11ff5a229bdfbcf9f13607"}
Nov 03 20:18:30.159238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.159173    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"]
Nov 03 20:18:30.164042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.164017    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-cwp8p"]
Nov 03 20:18:30.262051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.262003    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:30.971674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:30.970596    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:31.148884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:31.148846    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" event={"ID":"de77257b-93d9-472b-a007-a6d199acf1e2","Type":"ContainerStarted","Data":"6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634"}
Nov 03 20:18:31.176212 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:31.176158    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podStartSLOduration=17.176144948 podStartE2EDuration="17.176144948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:31.171114924 +0000 UTC m=+2303.840316395" watchObservedRunningTime="2026-11-03 20:18:31.176144948 +0000 UTC m=+2303.845346417"
Nov 03 20:18:31.459360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:31.459326    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32c977e2-140f-4591-9f8c-78361d36ef0f" path="/var/lib/kubelet/pods/32c977e2-140f-4591-9f8c-78361d36ef0f/volumes"
Nov 03 20:18:31.617243 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:18:31.617177    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:18:31.669410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:31.669373    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:31.676763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:31.676734    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:31.732422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:31.722077    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:32.137051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:32.136967    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"]
Nov 03 20:18:32.161992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:32.161907    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:32.163241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:32.163151    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:18:32.241802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:32.241744    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:32.903218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:32.900732    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:32.932295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:32.932253    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"]
Nov 03 20:18:33.021476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.021439    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"]
Nov 03 20:18:33.021669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.021572    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.030146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.030117    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:33.030369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.030251    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:18:33.130907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.130872    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.131084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.130993    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.131453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.131428    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p524j\" (UniqueName: \"kubernetes.io/projected/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-kube-api-access-p524j\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.232911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.232881    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p524j\" (UniqueName: \"kubernetes.io/projected/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-kube-api-access-p524j\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.233409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.232929    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.233409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.232995    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.233701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.233673    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.235416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.235393    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.240153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.240133    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p524j\" (UniqueName: \"kubernetes.io/projected/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-kube-api-access-p524j\") pod \"maas-api-revoke-sub-70dd62a7-m2h92\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.342058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.342023    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:33.513646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.513551    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:33.694499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.694461    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"]
Nov 03 20:18:33.695508 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:18:33.695475    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ee105b_78e0_4c9c_8923_29bb0d2aa102.slice/crio-835978c104e79236f598eff9e9621c036d5eb3d6d334371ed04782eecc2448bc WatchSource:0}: Error finding container 835978c104e79236f598eff9e9621c036d5eb3d6d334371ed04782eecc2448bc: Status 404 returned error can't find the container with id 835978c104e79236f598eff9e9621c036d5eb3d6d334371ed04782eecc2448bc
Nov 03 20:18:33.713651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.713626    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:33.922054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:33.922022    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:34.181001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.180893    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92" event={"ID":"c3ee105b-78e0-4c9c-8923-29bb0d2aa102","Type":"ContainerStarted","Data":"583feceaaca1d3d4c8034b9566ff1d255abf4ed93b268a0abbadcc54d5706458"}
Nov 03 20:18:34.181001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.180965    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92" event={"ID":"c3ee105b-78e0-4c9c-8923-29bb0d2aa102","Type":"ContainerStarted","Data":"835978c104e79236f598eff9e9621c036d5eb3d6d334371ed04782eecc2448bc"}
Nov 03 20:18:34.198255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.198168    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92" podStartSLOduration=2.198154995 podStartE2EDuration="2.198154995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:34.196648396 +0000 UTC m=+2306.865849869" watchObservedRunningTime="2026-11-03 20:18:34.198154995 +0000 UTC m=+2306.867356479"
Nov 03 20:18:34.585605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.585573    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:34.954351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.954237    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:34.956098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.956070    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"]
Nov 03 20:18:34.962237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.962215    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:34.970229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.967555    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a7343\""
Nov 03 20:18:34.970229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.967602    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a7343-dockercfg-7jrmp\""
Nov 03 20:18:34.970968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:34.970944    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"]
Nov 03 20:18:35.156753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.156713    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca057d53-559b-4e56-9635-24ef10b0b325-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:35.157174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.157139    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjzgx\" (UniqueName: \"kubernetes.io/projected/ca057d53-559b-4e56-9635-24ef10b0b325-kube-api-access-fjzgx\") pod \"payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:35.199259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.186457    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:35.267217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.263173    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca057d53-559b-4e56-9635-24ef10b0b325-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:35.279231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.276489    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjzgx\" (UniqueName: \"kubernetes.io/projected/ca057d53-559b-4e56-9635-24ef10b0b325-kube-api-access-fjzgx\") pod \"payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:35.279231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.264571    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca057d53-559b-4e56-9635-24ef10b0b325-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:35.311221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.306500    5282 generic.go:356] "Generic (PLEG): container finished" podID="c3ee105b-78e0-4c9c-8923-29bb0d2aa102" containerID="583feceaaca1d3d4c8034b9566ff1d255abf4ed93b268a0abbadcc54d5706458" exitCode=0
Nov 03 20:18:35.311221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.306715    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92" event={"ID":"c3ee105b-78e0-4c9c-8923-29bb0d2aa102","Type":"ContainerDied","Data":"583feceaaca1d3d4c8034b9566ff1d255abf4ed93b268a0abbadcc54d5706458"}
Nov 03 20:18:35.355222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.349667    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjzgx\" (UniqueName: \"kubernetes.io/projected/ca057d53-559b-4e56-9635-24ef10b0b325-kube-api-access-fjzgx\") pod \"payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:35.582461 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.582389    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:35.743946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.743912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:35.885504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.885410    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:35.905256 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:18:35.905171    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca057d53_559b_4e56_9635_24ef10b0b325.slice/crio-5b0072ee42eb8cec0a8d695b9c8eb29e054fd89b7c41315698bb1f6cffc9eb7f WatchSource:0}: Error finding container 5b0072ee42eb8cec0a8d695b9c8eb29e054fd89b7c41315698bb1f6cffc9eb7f: Status 404 returned error can't find the container with id 5b0072ee42eb8cec0a8d695b9c8eb29e054fd89b7c41315698bb1f6cffc9eb7f
Nov 03 20:18:35.906748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:35.906723    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"]
Nov 03 20:18:36.331053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.331012    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" event={"ID":"ca057d53-559b-4e56-9635-24ef10b0b325","Type":"ContainerStarted","Data":"f80cdbc0aadd09a85a75dba987b8865421f4f6dc5e36f28fdab7bc8d4c3b8133"}
Nov 03 20:18:36.331279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.331096    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" event={"ID":"ca057d53-559b-4e56-9635-24ef10b0b325","Type":"ContainerStarted","Data":"5b0072ee42eb8cec0a8d695b9c8eb29e054fd89b7c41315698bb1f6cffc9eb7f"}
Nov 03 20:18:36.331391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.331318    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:36.352526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.352470    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" podStartSLOduration=2.352452861 podStartE2EDuration="2.352452861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:36.347293004 +0000 UTC m=+2309.016494490" watchObservedRunningTime="2026-11-03 20:18:36.352452861 +0000 UTC m=+2309.021654333"
Nov 03 20:18:36.639914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.639886    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:36.681494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.681463    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:36.757226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.741920    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-openshift-ca\") pod \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") "
Nov 03 20:18:36.757226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.741980    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-kube-api-access-p524j\" (UniqueName: \"kubernetes.io/projected/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-kube-api-access-p524j\") pod \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") "
Nov 03 20:18:36.757226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.742102    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-xks-ca\") pod \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\" (UID: \"c3ee105b-78e0-4c9c-8923-29bb0d2aa102\") "
Nov 03 20:18:36.757226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.753881    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-xks-ca" pod "c3ee105b-78e0-4c9c-8923-29bb0d2aa102" (UID: "c3ee105b-78e0-4c9c-8923-29bb0d2aa102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:36.757226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.754350    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-openshift-ca" pod "c3ee105b-78e0-4c9c-8923-29bb0d2aa102" (UID: "c3ee105b-78e0-4c9c-8923-29bb0d2aa102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:36.819295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.815423    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-kube-api-access-p524j" pod "c3ee105b-78e0-4c9c-8923-29bb0d2aa102" (UID: "c3ee105b-78e0-4c9c-8923-29bb0d2aa102"). InnerVolumeSpecName "kube-api-access-p524j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:36.856308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.856058    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:36.856308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.856100    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p524j\" (UniqueName: \"kubernetes.io/projected/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-kube-api-access-p524j\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:36.856308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:36.856115    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ee105b-78e0-4c9c-8923-29bb0d2aa102-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:37.341598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:37.341562    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"
Nov 03 20:18:37.341598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:37.341585    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92" event={"ID":"c3ee105b-78e0-4c9c-8923-29bb0d2aa102","Type":"ContainerDied","Data":"835978c104e79236f598eff9e9621c036d5eb3d6d334371ed04782eecc2448bc"}
Nov 03 20:18:37.341814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:37.341637    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="835978c104e79236f598eff9e9621c036d5eb3d6d334371ed04782eecc2448bc"
Nov 03 20:18:37.399335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:37.399302    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:38.069304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:38.069271    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:38.856961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:38.856926    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:39.774523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:39.774492    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:40.487710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:40.487674    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:41.205393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:41.205364    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:42.065346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:42.056585    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:42.676364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:42.676333    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:42.992287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:42.992243    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"]
Nov 03 20:18:43.133221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.124951    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:43.514833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.514796    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"]
Nov 03 20:18:43.515809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.515793    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c3ee105b-78e0-4c9c-8923-29bb0d2aa102" containerName="revoke-keys"
Nov 03 20:18:43.515898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.515810    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ee105b-78e0-4c9c-8923-29bb0d2aa102" containerName="revoke-keys"
Nov 03 20:18:43.515968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.515959    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ee105b-78e0-4c9c-8923-29bb0d2aa102" containerName="revoke-keys"
Nov 03 20:18:43.526846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.526812    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"]
Nov 03 20:18:43.527027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.526940    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.546046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.546014    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:43.547709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.547684    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:18:43.614880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.614839    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:43.677137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.677088    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.709478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.709439    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vtp4\" (UniqueName: \"kubernetes.io/projected/2fbc971b-8d78-4467-8e2b-a74501f09f6e-kube-api-access-6vtp4\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.719289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.711055    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.821462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.821368    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.821640 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.821468    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vtp4\" (UniqueName: \"kubernetes.io/projected/2fbc971b-8d78-4467-8e2b-a74501f09f6e-kube-api-access-6vtp4\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.821708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.821671    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.822279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.822254    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.824052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.824022    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.829613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.829590    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vtp4\" (UniqueName: \"kubernetes.io/projected/2fbc971b-8d78-4467-8e2b-a74501f09f6e-kube-api-access-6vtp4\") pod \"maas-api-revoke-sub-ae0bc615-zskqt\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:43.865294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:43.865271    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:44.163001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:44.162279    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"]
Nov 03 20:18:44.424027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:44.423944    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt" event={"ID":"2fbc971b-8d78-4467-8e2b-a74501f09f6e","Type":"ContainerStarted","Data":"67e2c05fe24f55b42ab4ffaf48f3cff5d819f9e7ffaf130ec3b60725c09f8767"}
Nov 03 20:18:44.652217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:44.648651    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:45.435366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:45.435332    5282 generic.go:356] "Generic (PLEG): container finished" podID="2fbc971b-8d78-4467-8e2b-a74501f09f6e" containerID="2f2ebc4a50e5d639bb2acd516d069b46758ef7dc7c492d251ee70a495104d76a" exitCode=0
Nov 03 20:18:45.435747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:45.435423    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt" event={"ID":"2fbc971b-8d78-4467-8e2b-a74501f09f6e","Type":"ContainerDied","Data":"2f2ebc4a50e5d639bb2acd516d069b46758ef7dc7c492d251ee70a495104d76a"}
Nov 03 20:18:45.519164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:45.519134    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:46.379927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:46.379896    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:46.563414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:46.563378    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"]
Nov 03 20:18:46.620004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:46.618352    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:47.025219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.024976    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"]
Nov 03 20:18:47.035878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.035849    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"]
Nov 03 20:18:47.036080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.036036    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.049251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.048536    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mbh5\" (UniqueName: \"kubernetes.io/projected/12a349b0-843b-4da9-8632-7ccae2f421aa-kube-api-access-2mbh5\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.049251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.048617    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.049251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.049006    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.117040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.117007    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:47.150230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.150180    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-xks-ca\") pod \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") "
Nov 03 20:18:47.150432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.150278    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fbc971b-8d78-4467-8e2b-a74501f09f6e-kube-api-access-6vtp4\" (UniqueName: \"kubernetes.io/projected/2fbc971b-8d78-4467-8e2b-a74501f09f6e-kube-api-access-6vtp4\") pod \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") "
Nov 03 20:18:47.150432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.150412    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-openshift-ca\") pod \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\" (UID: \"2fbc971b-8d78-4467-8e2b-a74501f09f6e\") "
Nov 03 20:18:47.150954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.150927    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-openshift-ca" pod "2fbc971b-8d78-4467-8e2b-a74501f09f6e" (UID: "2fbc971b-8d78-4467-8e2b-a74501f09f6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:47.151247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.151220    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.151400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.151376    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbh5\" (UniqueName: \"kubernetes.io/projected/12a349b0-843b-4da9-8632-7ccae2f421aa-kube-api-access-2mbh5\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.151543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.151453    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.151657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.151639    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.151878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.151857    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.152777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.152748    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-xks-ca" pod "2fbc971b-8d78-4467-8e2b-a74501f09f6e" (UID: "2fbc971b-8d78-4467-8e2b-a74501f09f6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:47.153159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.153129    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbc971b-8d78-4467-8e2b-a74501f09f6e-kube-api-access-6vtp4" pod "2fbc971b-8d78-4467-8e2b-a74501f09f6e" (UID: "2fbc971b-8d78-4467-8e2b-a74501f09f6e"). InnerVolumeSpecName "kube-api-access-6vtp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:47.154455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.154434    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.161401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.161377    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbh5\" (UniqueName: \"kubernetes.io/projected/12a349b0-843b-4da9-8632-7ccae2f421aa-kube-api-access-2mbh5\") pod \"maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.193129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.193088    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" containerID="cri-o://535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89" gracePeriod=45
Nov 03 20:18:47.253748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.253711    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fbc971b-8d78-4467-8e2b-a74501f09f6e-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.253909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.253752    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vtp4\" (UniqueName: \"kubernetes.io/projected/2fbc971b-8d78-4467-8e2b-a74501f09f6e-kube-api-access-6vtp4\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.337656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.337616    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.139:8000/ready\": dial tcp 10.133.0.139:8000: connect: connection refused"
Nov 03 20:18:47.344797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.344773    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:18:47.368537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.368276    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:47.522852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.522825    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:18:47.559412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.559364    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-model-cache\") pod \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") "
Nov 03 20:18:47.559619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.559544    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-dshm\") pod \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") "
Nov 03 20:18:47.559619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.559595    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tmp-dir\") pod \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") "
Nov 03 20:18:47.559760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.559643    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05d5324-4bed-47b0-8dbd-3af0168a51b4-kube-api-access-dqwgf\" (UniqueName: \"kubernetes.io/projected/b05d5324-4bed-47b0-8dbd-3af0168a51b4-kube-api-access-dqwgf\") pod \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") "
Nov 03 20:18:47.559760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.559735    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-home\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-home\") pod \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") "
Nov 03 20:18:47.560130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.560094    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tls-certs\") pod \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\" (UID: \"b05d5324-4bed-47b0-8dbd-3af0168a51b4\") "
Nov 03 20:18:47.562536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.562507    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-home" pod "b05d5324-4bed-47b0-8dbd-3af0168a51b4" (UID: "b05d5324-4bed-47b0-8dbd-3af0168a51b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:47.562680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.562655    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tmp-dir" pod "b05d5324-4bed-47b0-8dbd-3af0168a51b4" (UID: "b05d5324-4bed-47b0-8dbd-3af0168a51b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:47.562774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.562751    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-model-cache" pod "b05d5324-4bed-47b0-8dbd-3af0168a51b4" (UID: "b05d5324-4bed-47b0-8dbd-3af0168a51b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:47.564377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.564314    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tls-certs" pod "b05d5324-4bed-47b0-8dbd-3af0168a51b4" (UID: "b05d5324-4bed-47b0-8dbd-3af0168a51b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:47.564717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.564664    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05d5324-4bed-47b0-8dbd-3af0168a51b4-kube-api-access-dqwgf" pod "b05d5324-4bed-47b0-8dbd-3af0168a51b4" (UID: "b05d5324-4bed-47b0-8dbd-3af0168a51b4"). InnerVolumeSpecName "kube-api-access-dqwgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:47.575617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.575585    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-dshm" pod "b05d5324-4bed-47b0-8dbd-3af0168a51b4" (UID: "b05d5324-4bed-47b0-8dbd-3af0168a51b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:47.576889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.576857    5282 generic.go:356] "Generic (PLEG): container finished" podID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerID="535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89" exitCode=0
Nov 03 20:18:47.576980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.576894    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" event={"ID":"b05d5324-4bed-47b0-8dbd-3af0168a51b4","Type":"ContainerDied","Data":"535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89"}
Nov 03 20:18:47.576980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.576947    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7" event={"ID":"b05d5324-4bed-47b0-8dbd-3af0168a51b4","Type":"ContainerDied","Data":"f3d890310a95265a0236c9b81e0da440f4ae0e3eb2eed27f9ffbcb93452f3012"}
Nov 03 20:18:47.576980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.576972    5282 scope.go:122] "RemoveContainer" containerID="535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89"
Nov 03 20:18:47.577157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.576908    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"
Nov 03 20:18:47.578981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.578960    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt" event={"ID":"2fbc971b-8d78-4467-8e2b-a74501f09f6e","Type":"ContainerDied","Data":"67e2c05fe24f55b42ab4ffaf48f3cff5d819f9e7ffaf130ec3b60725c09f8767"}
Nov 03 20:18:47.579072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.579005    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67e2c05fe24f55b42ab4ffaf48f3cff5d819f9e7ffaf130ec3b60725c09f8767"
Nov 03 20:18:47.579072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.579063    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"
Nov 03 20:18:47.590138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.589981    5282 scope.go:122] "RemoveContainer" containerID="535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89"
Nov 03 20:18:47.590414 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:18:47.590377    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89\": container with ID starting with 535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89 not found: ID does not exist" containerID="535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89"
Nov 03 20:18:47.590538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.590414    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89"} err="failed to get container status \"535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89\": rpc error: code = NotFound desc = could not find container \"535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89\": container with ID starting with 535d1bf687b39e1010f63be2be695d8810f5dab2aab5b53239eba9fc101c3e89 not found: ID does not exist"
Nov 03 20:18:47.607943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.607909    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"]
Nov 03 20:18:47.612895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.612862    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-49fb2b/test-model-49fb2b-kserve-6b4958b7fc-52tt7"]
Nov 03 20:18:47.662765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.662735    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.662765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.662764    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.663036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.662775    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.663036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.662785    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.663036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.662794    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqwgf\" (UniqueName: \"kubernetes.io/projected/b05d5324-4bed-47b0-8dbd-3af0168a51b4-kube-api-access-dqwgf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.663036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.662802    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b05d5324-4bed-47b0-8dbd-3af0168a51b4-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:47.671258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.671234    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:47.699221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:47.681002    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"]
Nov 03 20:18:48.613289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:48.606975    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z" event={"ID":"12a349b0-843b-4da9-8632-7ccae2f421aa","Type":"ContainerStarted","Data":"c2ed107193cdd179ea316b91d8dc501136c472c3a4bd93bcfd39cffd78d2952d"}
Nov 03 20:18:48.613289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:48.607095    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z" event={"ID":"12a349b0-843b-4da9-8632-7ccae2f421aa","Type":"ContainerStarted","Data":"fe98844da2fab81fd4544a758ca34fdc766891d6d2d8109a822272c452ef8be2"}
Nov 03 20:18:48.636406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:48.636336    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z" podStartSLOduration=2.636317927 podStartE2EDuration="2.636317927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:48.625511191 +0000 UTC m=+2321.294712662" watchObservedRunningTime="2026-11-03 20:18:48.636317927 +0000 UTC m=+2321.305519406"
Nov 03 20:18:48.833743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:48.833716    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:49.459626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:49.459598    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" path="/var/lib/kubelet/pods/b05d5324-4bed-47b0-8dbd-3af0168a51b4/volumes"
Nov 03 20:18:49.616789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:49.616755    5282 generic.go:356] "Generic (PLEG): container finished" podID="12a349b0-843b-4da9-8632-7ccae2f421aa" containerID="c2ed107193cdd179ea316b91d8dc501136c472c3a4bd93bcfd39cffd78d2952d" exitCode=0
Nov 03 20:18:49.617179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:49.616838    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z" event={"ID":"12a349b0-843b-4da9-8632-7ccae2f421aa","Type":"ContainerDied","Data":"c2ed107193cdd179ea316b91d8dc501136c472c3a4bd93bcfd39cffd78d2952d"}
Nov 03 20:18:49.861221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:49.855981    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:50.861976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:50.861950    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:51.009626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.009593    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-openshift-ca\") pod \"12a349b0-843b-4da9-8632-7ccae2f421aa\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") "
Nov 03 20:18:51.009832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.009695    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a349b0-843b-4da9-8632-7ccae2f421aa-kube-api-access-2mbh5\" (UniqueName: \"kubernetes.io/projected/12a349b0-843b-4da9-8632-7ccae2f421aa-kube-api-access-2mbh5\") pod \"12a349b0-843b-4da9-8632-7ccae2f421aa\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") "
Nov 03 20:18:51.009979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.009964    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-xks-ca\") pod \"12a349b0-843b-4da9-8632-7ccae2f421aa\" (UID: \"12a349b0-843b-4da9-8632-7ccae2f421aa\") "
Nov 03 20:18:51.010038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.009971    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-openshift-ca" pod "12a349b0-843b-4da9-8632-7ccae2f421aa" (UID: "12a349b0-843b-4da9-8632-7ccae2f421aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:51.010873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.010853    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:51.012455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.012426    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-xks-ca" pod "12a349b0-843b-4da9-8632-7ccae2f421aa" (UID: "12a349b0-843b-4da9-8632-7ccae2f421aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:51.012577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.012459    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a349b0-843b-4da9-8632-7ccae2f421aa-kube-api-access-2mbh5" pod "12a349b0-843b-4da9-8632-7ccae2f421aa" (UID: "12a349b0-843b-4da9-8632-7ccae2f421aa"). InnerVolumeSpecName "kube-api-access-2mbh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:51.071166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.071131    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:51.114216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.113796    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mbh5\" (UniqueName: \"kubernetes.io/projected/12a349b0-843b-4da9-8632-7ccae2f421aa-kube-api-access-2mbh5\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:51.114216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.113834    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a349b0-843b-4da9-8632-7ccae2f421aa-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:51.636726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.636696    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"
Nov 03 20:18:51.636909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.636726    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z" event={"ID":"12a349b0-843b-4da9-8632-7ccae2f421aa","Type":"ContainerDied","Data":"fe98844da2fab81fd4544a758ca34fdc766891d6d2d8109a822272c452ef8be2"}
Nov 03 20:18:51.636909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.636783    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe98844da2fab81fd4544a758ca34fdc766891d6d2d8109a822272c452ef8be2"
Nov 03 20:18:51.807142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:51.807112    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:52.601568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:52.601532    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:53.399775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:53.399737    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:54.102807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:54.102773    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:54.872625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:54.872585    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:54.994233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:54.993620    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:55.139312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:55.126970    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:56.061254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.056119    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:56.635945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.634670    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:56.737767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.737722    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" podUID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": context deadline exceeded"
Nov 03 20:18:56.759225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.757190    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.777902    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"]
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779403    5282 cpu_manager.go:414] "containerMap: removing container" podUID="2fbc971b-8d78-4467-8e2b-a74501f09f6e" containerName="revoke-keys"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779426    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fbc971b-8d78-4467-8e2b-a74501f09f6e" containerName="revoke-keys"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779477    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779487    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779517    5282 cpu_manager.go:414] "containerMap: removing container" podUID="12a349b0-843b-4da9-8632-7ccae2f421aa" containerName="revoke-keys"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779525    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a349b0-843b-4da9-8632-7ccae2f421aa" containerName="revoke-keys"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779754    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fbc971b-8d78-4467-8e2b-a74501f09f6e" containerName="revoke-keys"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779778    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05d5324-4bed-47b0-8dbd-3af0168a51b4" containerName="main"
Nov 03 20:18:56.782221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.779791    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a349b0-843b-4da9-8632-7ccae2f421aa" containerName="revoke-keys"
Nov 03 20:18:56.881970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.881905    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Nov 03 20:18:56.929159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.929069    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:56.961334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.961305    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"]
Nov 03 20:18:56.961637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.961622    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:56.969851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.969826    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:18:56.974702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:56.974677    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:18:57.051264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.051227    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.051632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.051602    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n2v\" (UniqueName: \"kubernetes.io/projected/b289275f-e02e-4411-a617-885bfa4cb876-kube-api-access-92n2v\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.052002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.051965    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.071658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.071626    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:57.155334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.155302    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.155674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.155647    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n2v\" (UniqueName: \"kubernetes.io/projected/b289275f-e02e-4411-a617-885bfa4cb876-kube-api-access-92n2v\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.155846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.155831    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.157672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.157639    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.162835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.158440    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.165359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.165331    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n2v\" (UniqueName: \"kubernetes.io/projected/b289275f-e02e-4411-a617-885bfa4cb876-kube-api-access-92n2v\") pod \"maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.192593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.192503    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:57.318847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.318512    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:18:57.530288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.530151    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:57.673279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:57.662586    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:58.024013 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:18:58.023984    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb289275f_e02e_4411_a617_885bfa4cb876.slice/crio-7f6c6ef2e9bdf3d67e8a2346018279bbfabdc960ae0e0e26268acd246465d6c8 WatchSource:0}: Error finding container 7f6c6ef2e9bdf3d67e8a2346018279bbfabdc960ae0e0e26268acd246465d6c8: Status 404 returned error can't find the container with id 7f6c6ef2e9bdf3d67e8a2346018279bbfabdc960ae0e0e26268acd246465d6c8
Nov 03 20:18:58.026830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.026801    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"]
Nov 03 20:18:58.026977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.026952    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main" containerID="cri-o://7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985" gracePeriod=45
Nov 03 20:18:58.128717 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:18:58.128684    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340a1e46_7aa9_423f_be41_4804a49a01d3.slice/crio-7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod340a1e46_7aa9_423f_be41_4804a49a01d3.slice/crio-conmon-7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:18:58.298380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.298326    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:58.446128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.446098    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:18:58.538772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.538739    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-model-cache\") pod \"340a1e46-7aa9-423f-be41-4804a49a01d3\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") "
Nov 03 20:18:58.538961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.538795    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-dshm\") pod \"340a1e46-7aa9-423f-be41-4804a49a01d3\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") "
Nov 03 20:18:58.538961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.538832    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-home\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-home\") pod \"340a1e46-7aa9-423f-be41-4804a49a01d3\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") "
Nov 03 20:18:58.539090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.539010    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-model-cache" pod "340a1e46-7aa9-423f-be41-4804a49a01d3" (UID: "340a1e46-7aa9-423f-be41-4804a49a01d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:58.539090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.539057    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/340a1e46-7aa9-423f-be41-4804a49a01d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/340a1e46-7aa9-423f-be41-4804a49a01d3-tls-certs\") pod \"340a1e46-7aa9-423f-be41-4804a49a01d3\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") "
Nov 03 20:18:58.539215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.539109    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-home" pod "340a1e46-7aa9-423f-be41-4804a49a01d3" (UID: "340a1e46-7aa9-423f-be41-4804a49a01d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:58.539275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.539251    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340a1e46-7aa9-423f-be41-4804a49a01d3-kube-api-access-x6v5z\" (UniqueName: \"kubernetes.io/projected/340a1e46-7aa9-423f-be41-4804a49a01d3-kube-api-access-x6v5z\") pod \"340a1e46-7aa9-423f-be41-4804a49a01d3\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") "
Nov 03 20:18:58.539449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.539429    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-tmp-dir\") pod \"340a1e46-7aa9-423f-be41-4804a49a01d3\" (UID: \"340a1e46-7aa9-423f-be41-4804a49a01d3\") "
Nov 03 20:18:58.539641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.539618    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-tmp-dir" pod "340a1e46-7aa9-423f-be41-4804a49a01d3" (UID: "340a1e46-7aa9-423f-be41-4804a49a01d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:58.540553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.540529    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:58.540553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.540557    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:58.540729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.540572    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:58.541988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.541964    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-dshm" pod "340a1e46-7aa9-423f-be41-4804a49a01d3" (UID: "340a1e46-7aa9-423f-be41-4804a49a01d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:58.542086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.542032    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/340a1e46-7aa9-423f-be41-4804a49a01d3-tls-certs" pod "340a1e46-7aa9-423f-be41-4804a49a01d3" (UID: "340a1e46-7aa9-423f-be41-4804a49a01d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:58.542150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.542111    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340a1e46-7aa9-423f-be41-4804a49a01d3-kube-api-access-x6v5z" pod "340a1e46-7aa9-423f-be41-4804a49a01d3" (UID: "340a1e46-7aa9-423f-be41-4804a49a01d3"). InnerVolumeSpecName "kube-api-access-x6v5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:58.642392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.642359    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/340a1e46-7aa9-423f-be41-4804a49a01d3-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:58.642392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.642387    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6v5z\" (UniqueName: \"kubernetes.io/projected/340a1e46-7aa9-423f-be41-4804a49a01d3-kube-api-access-x6v5z\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:58.642392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:58.642398    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/340a1e46-7aa9-423f-be41-4804a49a01d3-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:59.019415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.019381    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf" event={"ID":"b289275f-e02e-4411-a617-885bfa4cb876","Type":"ContainerStarted","Data":"a4a5bb99ba6702dd3ad9f92a142685564e411416a93e0cda8016b998a76075c1"}
Nov 03 20:18:59.019911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.019439    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf" event={"ID":"b289275f-e02e-4411-a617-885bfa4cb876","Type":"ContainerStarted","Data":"7f6c6ef2e9bdf3d67e8a2346018279bbfabdc960ae0e0e26268acd246465d6c8"}
Nov 03 20:18:59.021087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.021061    5282 generic.go:356] "Generic (PLEG): container finished" podID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerID="7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985" exitCode=0
Nov 03 20:18:59.021243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.021094    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"
Nov 03 20:18:59.021243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.021137    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" event={"ID":"340a1e46-7aa9-423f-be41-4804a49a01d3","Type":"ContainerDied","Data":"7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985"}
Nov 03 20:18:59.021243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.021172    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb" event={"ID":"340a1e46-7aa9-423f-be41-4804a49a01d3","Type":"ContainerDied","Data":"5da2bfb9527aa76a7942c3d862ac9ab80db63ea8c577fe9bfb2764529706a022"}
Nov 03 20:18:59.021243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.021230    5282 scope.go:122] "RemoveContainer" containerID="7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985"
Nov 03 20:18:59.035732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.035712    5282 scope.go:122] "RemoveContainer" containerID="7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985"
Nov 03 20:18:59.036006 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:18:59.035984    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985\": container with ID starting with 7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985 not found: ID does not exist" containerID="7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985"
Nov 03 20:18:59.036076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.036017    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985"} err="failed to get container status \"7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985\": rpc error: code = NotFound desc = could not find container \"7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985\": container with ID starting with 7af9f11d18bd461dd482a88f1fa7e3605acae855fe803a5b7febe682813c2985 not found: ID does not exist"
Nov 03 20:18:59.039634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.039589    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf" podStartSLOduration=3.039563093 podStartE2EDuration="3.039563093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:59.036638343 +0000 UTC m=+2331.705839814" watchObservedRunningTime="2026-11-03 20:18:59.039563093 +0000 UTC m=+2331.708764571"
Nov 03 20:18:59.055489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.055455    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"]
Nov 03 20:18:59.060460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.060435    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-43b780/test-model-43b780-kserve-5b68c65c59-lchnb"]
Nov 03 20:18:59.086414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.086379    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:59.463655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.463563    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" path="/var/lib/kubelet/pods/340a1e46-7aa9-423f-be41-4804a49a01d3/volumes"
Nov 03 20:18:59.577720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.577639    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"]
Nov 03 20:18:59.577993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.577963    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z" podUID="95d58810-f6b1-43cf-893f-37bec467763b" containerName="maas-api" containerID="cri-o://4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c" gracePeriod=30
Nov 03 20:18:59.640311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.640281    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:59.801682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.801507    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:18:59.929291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:18:59.928369    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:00.258217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.252362    5282 generic.go:356] "Generic (PLEG): container finished" podID="b289275f-e02e-4411-a617-885bfa4cb876" containerID="a4a5bb99ba6702dd3ad9f92a142685564e411416a93e0cda8016b998a76075c1" exitCode=0
Nov 03 20:19:00.258217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.252662    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf" event={"ID":"b289275f-e02e-4411-a617-885bfa4cb876","Type":"ContainerDied","Data":"a4a5bb99ba6702dd3ad9f92a142685564e411416a93e0cda8016b998a76075c1"}
Nov 03 20:19:00.295221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.279885    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:00.339684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.339647    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"]
Nov 03 20:19:00.472281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.472243    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49fb2b-prb9z"]
Nov 03 20:19:00.500782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.500738    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:00.668775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.668748    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:19:00.672187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.672164    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d58810-f6b1-43cf-893f-37bec467763b-kube-api-access-qnzd9\" (UniqueName: \"kubernetes.io/projected/95d58810-f6b1-43cf-893f-37bec467763b-kube-api-access-qnzd9\") pod \"95d58810-f6b1-43cf-893f-37bec467763b\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") "
Nov 03 20:19:00.672364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.672225    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d58810-f6b1-43cf-893f-37bec467763b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d58810-f6b1-43cf-893f-37bec467763b-openshift-service-ca\") pod \"95d58810-f6b1-43cf-893f-37bec467763b\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") "
Nov 03 20:19:00.672364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.672301    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-metrics-certs\") pod \"95d58810-f6b1-43cf-893f-37bec467763b\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") "
Nov 03 20:19:00.672612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.672589    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-maas-api-tls\") pod \"95d58810-f6b1-43cf-893f-37bec467763b\" (UID: \"95d58810-f6b1-43cf-893f-37bec467763b\") "
Nov 03 20:19:00.672731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.672583    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d58810-f6b1-43cf-893f-37bec467763b-openshift-service-ca" pod "95d58810-f6b1-43cf-893f-37bec467763b" (UID: "95d58810-f6b1-43cf-893f-37bec467763b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:00.673470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.673448    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d58810-f6b1-43cf-893f-37bec467763b-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:00.674748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.674714    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d58810-f6b1-43cf-893f-37bec467763b-kube-api-access-qnzd9" pod "95d58810-f6b1-43cf-893f-37bec467763b" (UID: "95d58810-f6b1-43cf-893f-37bec467763b"). InnerVolumeSpecName "kube-api-access-qnzd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:00.674969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.674942    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-metrics-certs" pod "95d58810-f6b1-43cf-893f-37bec467763b" (UID: "95d58810-f6b1-43cf-893f-37bec467763b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:00.674969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.674957    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-maas-api-tls" pod "95d58810-f6b1-43cf-893f-37bec467763b" (UID: "95d58810-f6b1-43cf-893f-37bec467763b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:00.703211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.702503    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"]
Nov 03 20:19:00.704071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.704001    5282 cpu_manager.go:414] "containerMap: removing container" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main"
Nov 03 20:19:00.704071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.704029    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main"
Nov 03 20:19:00.704381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.704118    5282 cpu_manager.go:414] "containerMap: removing container" podUID="95d58810-f6b1-43cf-893f-37bec467763b" containerName="maas-api"
Nov 03 20:19:00.704381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.704127    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d58810-f6b1-43cf-893f-37bec467763b" containerName="maas-api"
Nov 03 20:19:00.704477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.704396    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d58810-f6b1-43cf-893f-37bec467763b" containerName="maas-api"
Nov 03 20:19:00.704477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.704422    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="340a1e46-7aa9-423f-be41-4804a49a01d3" containerName="main"
Nov 03 20:19:00.725306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.713315    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.725306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.716965    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0a7343-serving-cert\""
Nov 03 20:19:00.725306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.717293    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"]
Nov 03 20:19:00.775794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.775676    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe48dd6d-6155-474b-9cbd-01bc9898b66f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.776145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.776085    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-metrics-certs\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.776145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.776125    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjth2\" (UniqueName: \"kubernetes.io/projected/fe48dd6d-6155-474b-9cbd-01bc9898b66f-kube-api-access-cjth2\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.776298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.776170    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.776417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.776396    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:00.776474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.776423    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnzd9\" (UniqueName: \"kubernetes.io/projected/95d58810-f6b1-43cf-893f-37bec467763b-kube-api-access-qnzd9\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:00.776474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.776441    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d58810-f6b1-43cf-893f-37bec467763b-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:00.878552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.878511    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-metrics-certs\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.878744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.878563    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjth2\" (UniqueName: \"kubernetes.io/projected/fe48dd6d-6155-474b-9cbd-01bc9898b66f-kube-api-access-cjth2\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.878744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.878605    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.878883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.878759    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe48dd6d-6155-474b-9cbd-01bc9898b66f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.879730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.879698    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe48dd6d-6155-474b-9cbd-01bc9898b66f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.881442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.881413    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-metrics-certs\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.881605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.881584    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:00.886872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:00.886842    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjth2\" (UniqueName: \"kubernetes.io/projected/fe48dd6d-6155-474b-9cbd-01bc9898b66f-kube-api-access-cjth2\") pod \"maas-api-e2e-inf-b-0a7343-57d64cf499-prttj\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:01.043687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.043602    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:01.178623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.178587    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:01.358907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.358655    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"]
Nov 03 20:19:01.391259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.391223    5282 generic.go:356] "Generic (PLEG): container finished" podID="95d58810-f6b1-43cf-893f-37bec467763b" containerID="4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c" exitCode=0
Nov 03 20:19:01.391379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.391264    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z" event={"ID":"95d58810-f6b1-43cf-893f-37bec467763b","Type":"ContainerDied","Data":"4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c"}
Nov 03 20:19:01.391379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.391335    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z" event={"ID":"95d58810-f6b1-43cf-893f-37bec467763b","Type":"ContainerDied","Data":"6bfbf36f3074b0c13c44b1e97cf0bd067525776c70592f31f52a8ab02df399c2"}
Nov 03 20:19:01.391379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.391353    5282 scope.go:122] "RemoveContainer" containerID="4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c"
Nov 03 20:19:01.391537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.391276    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"
Nov 03 20:19:01.392772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.392742    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj" event={"ID":"fe48dd6d-6155-474b-9cbd-01bc9898b66f","Type":"ContainerStarted","Data":"bbfca0a5706ea9f809d5b0989826a67f9cf582d4d1cccf588e3bbfa1d990d7b6"}
Nov 03 20:19:01.409532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.409507    5282 scope.go:122] "RemoveContainer" containerID="4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c"
Nov 03 20:19:01.409898 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:01.409870    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c\": container with ID starting with 4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c not found: ID does not exist" containerID="4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c"
Nov 03 20:19:01.409989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.409910    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c"} err="failed to get container status \"4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c\": rpc error: code = NotFound desc = could not find container \"4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c\": container with ID starting with 4339d59647c18e28fdda2cd90a8a46e80093c3ea3d28e36daafcbc64312d209c not found: ID does not exist"
Nov 03 20:19:01.429862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.429831    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"]
Nov 03 20:19:01.436621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.436591    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-49fb2b-bd4dc4b75-2zn7z"]
Nov 03 20:19:01.463016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.462974    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a349b0-843b-4da9-8632-7ccae2f421aa" path="/var/lib/kubelet/pods/12a349b0-843b-4da9-8632-7ccae2f421aa/volumes"
Nov 03 20:19:01.463662 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.463639    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d58810-f6b1-43cf-893f-37bec467763b" path="/var/lib/kubelet/pods/95d58810-f6b1-43cf-893f-37bec467763b/volumes"
Nov 03 20:19:01.548697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.548673    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:19:01.608449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.608358    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-openshift-ca\") pod \"b289275f-e02e-4411-a617-885bfa4cb876\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") "
Nov 03 20:19:01.608617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.608548    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-xks-ca\") pod \"b289275f-e02e-4411-a617-885bfa4cb876\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") "
Nov 03 20:19:01.608972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.608824    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-openshift-ca" pod "b289275f-e02e-4411-a617-885bfa4cb876" (UID: "b289275f-e02e-4411-a617-885bfa4cb876"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:01.609126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.609102    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b289275f-e02e-4411-a617-885bfa4cb876-kube-api-access-92n2v\" (UniqueName: \"kubernetes.io/projected/b289275f-e02e-4411-a617-885bfa4cb876-kube-api-access-92n2v\") pod \"b289275f-e02e-4411-a617-885bfa4cb876\" (UID: \"b289275f-e02e-4411-a617-885bfa4cb876\") "
Nov 03 20:19:01.610432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.610409    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:01.611169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.611146    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-xks-ca" pod "b289275f-e02e-4411-a617-885bfa4cb876" (UID: "b289275f-e02e-4411-a617-885bfa4cb876"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:01.611505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.611484    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b289275f-e02e-4411-a617-885bfa4cb876-kube-api-access-92n2v" pod "b289275f-e02e-4411-a617-885bfa4cb876" (UID: "b289275f-e02e-4411-a617-885bfa4cb876"). InnerVolumeSpecName "kube-api-access-92n2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:01.613556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.613526    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" containerID="cri-o://1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79" gracePeriod=45
Nov 03 20:19:01.630209 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:01.630156    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:19:01.709215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.709165    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": dial tcp 10.133.0.140:8000: connect: connection refused"
Nov 03 20:19:01.712629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.712601    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b289275f-e02e-4411-a617-885bfa4cb876-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:01.712756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.712632    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92n2v\" (UniqueName: \"kubernetes.io/projected/b289275f-e02e-4411-a617-885bfa4cb876-kube-api-access-92n2v\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:01.940833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:01.940801    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:19:02.018908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.018851    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-dshm\") pod \"fb501b02-fb73-487a-ad94-08222d91812c\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") "
Nov 03 20:19:02.019296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.018924    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-tmp-dir\") pod \"fb501b02-fb73-487a-ad94-08222d91812c\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") "
Nov 03 20:19:02.019296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.019017    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb501b02-fb73-487a-ad94-08222d91812c-kube-api-access-lmhs5\" (UniqueName: \"kubernetes.io/projected/fb501b02-fb73-487a-ad94-08222d91812c-kube-api-access-lmhs5\") pod \"fb501b02-fb73-487a-ad94-08222d91812c\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") "
Nov 03 20:19:02.019296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.019122    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-model-cache\") pod \"fb501b02-fb73-487a-ad94-08222d91812c\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") "
Nov 03 20:19:02.019548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.019308    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-home\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-home\") pod \"fb501b02-fb73-487a-ad94-08222d91812c\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") "
Nov 03 20:19:02.019548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.019516    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb501b02-fb73-487a-ad94-08222d91812c-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb501b02-fb73-487a-ad94-08222d91812c-tls-certs\") pod \"fb501b02-fb73-487a-ad94-08222d91812c\" (UID: \"fb501b02-fb73-487a-ad94-08222d91812c\") "
Nov 03 20:19:02.021480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.021437    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-tmp-dir" pod "fb501b02-fb73-487a-ad94-08222d91812c" (UID: "fb501b02-fb73-487a-ad94-08222d91812c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:02.021822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.021797    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-home" pod "fb501b02-fb73-487a-ad94-08222d91812c" (UID: "fb501b02-fb73-487a-ad94-08222d91812c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:02.022002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.021983    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-model-cache" pod "fb501b02-fb73-487a-ad94-08222d91812c" (UID: "fb501b02-fb73-487a-ad94-08222d91812c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:02.023926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.023903    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-dshm" pod "fb501b02-fb73-487a-ad94-08222d91812c" (UID: "fb501b02-fb73-487a-ad94-08222d91812c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:02.024115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.024089    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb501b02-fb73-487a-ad94-08222d91812c-kube-api-access-lmhs5" pod "fb501b02-fb73-487a-ad94-08222d91812c" (UID: "fb501b02-fb73-487a-ad94-08222d91812c"). InnerVolumeSpecName "kube-api-access-lmhs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:02.028407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.028379    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb501b02-fb73-487a-ad94-08222d91812c-tls-certs" pod "fb501b02-fb73-487a-ad94-08222d91812c" (UID: "fb501b02-fb73-487a-ad94-08222d91812c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:02.087500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.087462    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:02.148278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.145884    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb501b02-fb73-487a-ad94-08222d91812c-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:02.148278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.145922    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:02.148278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.145939    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:02.148278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.145952    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmhs5\" (UniqueName: \"kubernetes.io/projected/fb501b02-fb73-487a-ad94-08222d91812c-kube-api-access-lmhs5\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:02.148278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.145969    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:02.148278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.145984    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb501b02-fb73-487a-ad94-08222d91812c-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:02.405790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.405701    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj" event={"ID":"fe48dd6d-6155-474b-9cbd-01bc9898b66f","Type":"ContainerStarted","Data":"d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb"}
Nov 03 20:19:02.405790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.405764    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:02.407361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.407341    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"
Nov 03 20:19:02.407545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.407399    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf" event={"ID":"b289275f-e02e-4411-a617-885bfa4cb876","Type":"ContainerDied","Data":"7f6c6ef2e9bdf3d67e8a2346018279bbfabdc960ae0e0e26268acd246465d6c8"}
Nov 03 20:19:02.407545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.407449    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f6c6ef2e9bdf3d67e8a2346018279bbfabdc960ae0e0e26268acd246465d6c8"
Nov 03 20:19:02.408988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.408966    5282 generic.go:356] "Generic (PLEG): container finished" podID="fb501b02-fb73-487a-ad94-08222d91812c" containerID="1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79" exitCode=0
Nov 03 20:19:02.409154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.409001    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"
Nov 03 20:19:02.409154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.409034    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" event={"ID":"fb501b02-fb73-487a-ad94-08222d91812c","Type":"ContainerDied","Data":"1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79"}
Nov 03 20:19:02.409154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.409076    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv" event={"ID":"fb501b02-fb73-487a-ad94-08222d91812c","Type":"ContainerDied","Data":"3ba77f1fb9fe16636d1564a1b3ec8da6d27f8f303ca836cba08f7a6003c41e7e"}
Nov 03 20:19:02.409154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.409098    5282 scope.go:122] "RemoveContainer" containerID="1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79"
Nov 03 20:19:02.422244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.422213    5282 scope.go:122] "RemoveContainer" containerID="1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79"
Nov 03 20:19:02.422509 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:02.422488    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79\": container with ID starting with 1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79 not found: ID does not exist" containerID="1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79"
Nov 03 20:19:02.422592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.422515    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79"} err="failed to get container status \"1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79\": rpc error: code = NotFound desc = could not find container \"1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79\": container with ID starting with 1230c59b2419b2aa0a785b2e2d8aa222c9a7f0d00497ffe5882d603bc0559e79 not found: ID does not exist"
Nov 03 20:19:02.426115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.426075    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj" podStartSLOduration=8.016182616 podStartE2EDuration="8.426061207s" totalImagesPullingTime="409.878591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:02.423582324 +0000 UTC m=+2335.092783794" watchObservedRunningTime="2026-11-03 20:19:02.426061207 +0000 UTC m=+2335.095262676"
Nov 03 20:19:02.447665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.447636    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"]
Nov 03 20:19:02.455528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.455499    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-facc79/test-model-facc79-kserve-ffcf49c85-tf5jv"]
Nov 03 20:19:02.992174 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:02.992138    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:03.460278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:03.460240    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb501b02-fb73-487a-ad94-08222d91812c" path="/var/lib/kubelet/pods/fb501b02-fb73-487a-ad94-08222d91812c/volumes"
Nov 03 20:19:03.475754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:03.475698    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": read tcp 10.133.0.2:37540->10.133.0.156:15021: read: connection reset by peer" start-of-body=
Nov 03 20:19:03.475869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:03.475797    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": read tcp 10.133.0.2:37540->10.133.0.156:15021: read: connection reset by peer"
Nov 03 20:19:03.696128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:03.696054    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:04.431426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:04.431393    5282 generic.go:356] "Generic (PLEG): container finished" podID="f47726d4-a756-464a-9fff-465b9ab355b0" containerID="9249a1cfd31aa77756189e5fe6182125ac13a461020fb11578ed6bb72ffc2a5b" exitCode=0
Nov 03 20:19:04.431609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:04.431445    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" event={"ID":"f47726d4-a756-464a-9fff-465b9ab355b0","Type":"ContainerDied","Data":"9249a1cfd31aa77756189e5fe6182125ac13a461020fb11578ed6bb72ffc2a5b"}
Nov 03 20:19:04.431832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:04.431819    5282 scope.go:122] "RemoveContainer" containerID="9249a1cfd31aa77756189e5fe6182125ac13a461020fb11578ed6bb72ffc2a5b"
Nov 03 20:19:04.433629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:04.433593    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:04.433735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:04.433666    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:04.433735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:04.433707    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:04.503756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:04.503722    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:05.286342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:05.286308    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:05.481134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:05.481096    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" event={"ID":"f47726d4-a756-464a-9fff-465b9ab355b0","Type":"ContainerStarted","Data":"8b941d55abe35057078c71f99a74fbc915cdc240d61079ade03fc40f2c4fad61"}
Nov 03 20:19:05.799429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:05.799391    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:19:05.799940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:05.799639    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:19:05.799940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:05.799696    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:19:06.012083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.012048    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:06.445504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.445471    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:06.570217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.563777    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:06.754343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.754290    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"]
Nov 03 20:19:06.754933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.754898    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" containerID="cri-o://05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf" gracePeriod=30
Nov 03 20:19:06.805616 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.805579    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:19:06.806263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.806226    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:06.806510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.806481    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:19:06.934588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:06.934551    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:07.049269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.049133    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:07.512272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.463535    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"]
Nov 03 20:19:07.570477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.566535    5282 cpu_manager.go:414] "containerMap: removing container" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main"
Nov 03 20:19:07.570477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.566566    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main"
Nov 03 20:19:07.570477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.566629    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b289275f-e02e-4411-a617-885bfa4cb876" containerName="revoke-keys"
Nov 03 20:19:07.570477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.566636    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b289275f-e02e-4411-a617-885bfa4cb876" containerName="revoke-keys"
Nov 03 20:19:07.570477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.566866    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b289275f-e02e-4411-a617-885bfa4cb876" containerName="revoke-keys"
Nov 03 20:19:07.570477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.566891    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb501b02-fb73-487a-ad94-08222d91812c" containerName="main"
Nov 03 20:19:07.583252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.583168    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.596491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.596350    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:19:07.602993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.602332    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zlwvw\""
Nov 03 20:19:07.633714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.633690    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"]
Nov 03 20:19:07.693256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.693216    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.693534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.693510    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd7x\" (UniqueName: \"kubernetes.io/projected/caea382a-8001-41e8-b21a-a5d4dbf3eeda-kube-api-access-jtd7x\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.694095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.694069    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.798568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.798472    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.798746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.798607    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.798746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.798651    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd7x\" (UniqueName: \"kubernetes.io/projected/caea382a-8001-41e8-b21a-a5d4dbf3eeda-kube-api-access-jtd7x\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.798746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.798667    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:19:07.798928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.798750    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:19:07.799525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.799497    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.801673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.801650    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.806205 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.806181    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd7x\" (UniqueName: \"kubernetes.io/projected/caea382a-8001-41e8-b21a-a5d4dbf3eeda-kube-api-access-jtd7x\") pod \"maas-api-revoke-keys-e2e-inf-b-facc79-24zjq\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:07.930721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:07.930686    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:08.002224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:08.001741    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:08.426028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:08.425952    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:19:08.464832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:08.464481    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"]
Nov 03 20:19:08.638566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:08.638532    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq" event={"ID":"caea382a-8001-41e8-b21a-a5d4dbf3eeda","Type":"ContainerStarted","Data":"c8b4515f1aa67a4d28952991d24753e3ba032582d00da2a59f3171ce9b70d57c"}
Nov 03 20:19:08.799066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:08.799017    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:19:08.799307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:08.799085    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:19:08.992892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:08.992863    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:09.152422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.152383    5282 scope.go:122] "RemoveContainer" containerID="4faf6b80264df00fbc8eb6992c4baebfaab3a835ae38f72d281e29bab6b5d6d3"
Nov 03 20:19:09.645185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.645146    5282 patch_prober.go:29] interesting pod/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:19:09.645394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.645248    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:19:09.649188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.649162    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq" event={"ID":"caea382a-8001-41e8-b21a-a5d4dbf3eeda","Type":"ContainerStarted","Data":"00a50bb2930928670beea3f4dd7e11ff204aad12574751befc5634bcf0f3808c"}
Nov 03 20:19:09.667586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.667543    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq" podStartSLOduration=2.667530896 podStartE2EDuration="2.667530896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:09.66436683 +0000 UTC m=+2342.333568305" watchObservedRunningTime="2026-11-03 20:19:09.667530896 +0000 UTC m=+2342.336732366"
Nov 03 20:19:09.798568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.798535    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:19:09.798751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.798603    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:19:09.899825 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:09.899752    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:10.574616 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:10.574580    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:10.675220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:10.672365    5282 generic.go:356] "Generic (PLEG): container finished" podID="caea382a-8001-41e8-b21a-a5d4dbf3eeda" containerID="00a50bb2930928670beea3f4dd7e11ff204aad12574751befc5634bcf0f3808c" exitCode=0
Nov 03 20:19:10.675220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:10.672473    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq" event={"ID":"caea382a-8001-41e8-b21a-a5d4dbf3eeda","Type":"ContainerDied","Data":"00a50bb2930928670beea3f4dd7e11ff204aad12574751befc5634bcf0f3808c"}
Nov 03 20:19:10.804444 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:10.804403    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:19:10.804717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:10.804691    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 20:19:11.211592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:11.211558    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:11.803098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:11.803014    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:19:11.804129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:11.804108    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:19:11.804766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:11.804747    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:19:11.893852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:11.893144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:11.938571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:11.938486    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:12.075223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.063184    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-xks-ca\") pod \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") "
Nov 03 20:19:12.075223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.063550    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caea382a-8001-41e8-b21a-a5d4dbf3eeda-kube-api-access-jtd7x\" (UniqueName: \"kubernetes.io/projected/caea382a-8001-41e8-b21a-a5d4dbf3eeda-kube-api-access-jtd7x\") pod \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") "
Nov 03 20:19:12.075223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.063712    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-openshift-ca\") pod \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\" (UID: \"caea382a-8001-41e8-b21a-a5d4dbf3eeda\") "
Nov 03 20:19:12.120551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.116679    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-xks-ca" pod "caea382a-8001-41e8-b21a-a5d4dbf3eeda" (UID: "caea382a-8001-41e8-b21a-a5d4dbf3eeda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:12.138125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.138084    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-openshift-ca" pod "caea382a-8001-41e8-b21a-a5d4dbf3eeda" (UID: "caea382a-8001-41e8-b21a-a5d4dbf3eeda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:12.153769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.151442    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.153769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.151474    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caea382a-8001-41e8-b21a-a5d4dbf3eeda-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.172641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.172583    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caea382a-8001-41e8-b21a-a5d4dbf3eeda-kube-api-access-jtd7x" pod "caea382a-8001-41e8-b21a-a5d4dbf3eeda" (UID: "caea382a-8001-41e8-b21a-a5d4dbf3eeda"). InnerVolumeSpecName "kube-api-access-jtd7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:12.257586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.257549    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd7x\" (UniqueName: \"kubernetes.io/projected/caea382a-8001-41e8-b21a-a5d4dbf3eeda-kube-api-access-jtd7x\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.257760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.257588    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:19:12.290350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.290321    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"]
Nov 03 20:19:12.291276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291259    5282 cpu_manager.go:414] "containerMap: removing container" podUID="caea382a-8001-41e8-b21a-a5d4dbf3eeda" containerName="revoke-keys"
Nov 03 20:19:12.291356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291277    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caea382a-8001-41e8-b21a-a5d4dbf3eeda" containerName="revoke-keys"
Nov 03 20:19:12.291356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291313    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy"
Nov 03 20:19:12.291356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291318    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy"
Nov 03 20:19:12.291356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291330    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy"
Nov 03 20:19:12.291356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291337    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy"
Nov 03 20:19:12.291618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291513    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="caea382a-8001-41e8-b21a-a5d4dbf3eeda" containerName="revoke-keys"
Nov 03 20:19:12.291618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291526    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy"
Nov 03 20:19:12.291618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.291534    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerName="istio-proxy"
Nov 03 20:19:12.295671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.295650    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.305860 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.305831    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"]
Nov 03 20:19:12.358748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.358718    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-token\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-token\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.358946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.358776    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-envoy\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.358946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.358803    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-certs\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.358946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.358924    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-kube-api-access-rmgm9\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.359127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.358954    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istiod-ca-cert\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.359127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359012    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-data\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.359127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359085    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-credential-socket\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.359308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359179    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-socket\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.359308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359269    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-podinfo\") pod \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\" (UID: \"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2\") "
Nov 03 20:19:12.359676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359646    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.359788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359695    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.359788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359762    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.359897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.359806    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.360074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360054    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istiod-ca-cert" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:12.360167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360116    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.360319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360298    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.360572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360551    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.360670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360455    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-credential-socket" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.360764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360628    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-socket" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.360764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360640    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-data" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.360891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360798    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm2fw\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-kube-api-access-nm2fw\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.360953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360876    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.360953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.360881    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-certs" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.361118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.361079    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.361118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.361103    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.361243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.361119    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.361243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.361134    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.361243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.361148    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.361673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.361644    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-envoy" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:12.361797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.361755    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-token" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:12.362975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.362949    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-kube-api-access-rmgm9" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "kube-api-access-rmgm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:12.378433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.378406    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-podinfo" pod "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" (UID: "ab761e2b-c2a0-4a41-a8ec-efdc9df389e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:12.432111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.432074    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"]
Nov 03 20:19:12.438419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.438397    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.451767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.451740    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"]
Nov 03 20:19:12.465524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465487    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.465693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465527    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.465693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465582    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.465693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465621    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.465693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465686    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.465922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465735    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmbm\" (UniqueName: \"kubernetes.io/projected/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-kube-api-access-xwmbm\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.465986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465939    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.465982    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466010    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466071    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm2fw\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-kube-api-access-nm2fw\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466108    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466134    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.466340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466282    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.466340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466305    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.466340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466319    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.466340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466334    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2-kube-api-access-rmgm9\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.466758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466605    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istiod-ca-cert\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466627    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-certs\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466871    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.466941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.466904    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-data\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.467126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.467103    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-credential-socket\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.469841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.469820    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-podinfo\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.469950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.469857    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-envoy\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.473887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.473859    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-token\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.474891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.474866    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm2fw\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-kube-api-access-nm2fw\") pod \"e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") " pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.567493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.567405    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.567723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.567702    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.567795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.567768    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmbm\" (UniqueName: \"kubernetes.io/projected/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-kube-api-access-xwmbm\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.568112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.568091    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.570052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.570033    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.576132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.576113    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmbm\" (UniqueName: \"kubernetes.io/projected/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-kube-api-access-xwmbm\") pod \"maas-api-revoke-sub-5c8f4714-ln9br\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.603229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.603185    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:12.631261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.631171    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:12.756436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.755959    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:12.760252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.760166    5282 generic.go:356] "Generic (PLEG): container finished" podID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" containerID="05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf" exitCode=0
Nov 03 20:19:12.760405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.760272    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" event={"ID":"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2","Type":"ContainerDied","Data":"05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf"}
Nov 03 20:19:12.760405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.760333    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv" event={"ID":"ab761e2b-c2a0-4a41-a8ec-efdc9df389e2","Type":"ContainerDied","Data":"effd585421a4dd6ba6d04eaf171f0321f06acc9562aef961790cc633473f392a"}
Nov 03 20:19:12.760405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.760360    5282 scope.go:122] "RemoveContainer" containerID="05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf"
Nov 03 20:19:12.761299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.761257    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"
Nov 03 20:19:12.768108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.768077    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"
Nov 03 20:19:12.770021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.769995    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq" event={"ID":"caea382a-8001-41e8-b21a-a5d4dbf3eeda","Type":"ContainerDied","Data":"c8b4515f1aa67a4d28952991d24753e3ba032582d00da2a59f3171ce9b70d57c"}
Nov 03 20:19:12.770188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.770048    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b4515f1aa67a4d28952991d24753e3ba032582d00da2a59f3171ce9b70d57c"
Nov 03 20:19:12.787669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.787645    5282 scope.go:122] "RemoveContainer" containerID="bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6"
Nov 03 20:19:12.796304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.796279    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"]
Nov 03 20:19:12.802499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.802474    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49fb2b-openshift-default-569f598477-7gzrv"]
Nov 03 20:19:12.860919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.860818    5282 scope.go:122] "RemoveContainer" containerID="05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf"
Nov 03 20:19:12.861478 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:12.861452    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf\": container with ID starting with 05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf not found: ID does not exist" containerID="05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf"
Nov 03 20:19:12.861574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.861490    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf"} err="failed to get container status \"05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf\": rpc error: code = NotFound desc = could not find container \"05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf\": container with ID starting with 05ec1dd41df2fda4ba41d5fb236f6ecccc5ecc87d6c0b4c33c672e3ca6bd12bf not found: ID does not exist"
Nov 03 20:19:12.861574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.861516    5282 scope.go:122] "RemoveContainer" containerID="bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6"
Nov 03 20:19:12.861843 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:12.861819    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6\": container with ID starting with bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6 not found: ID does not exist" containerID="bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6"
Nov 03 20:19:12.861937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:12.861854    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6"} err="failed to get container status \"bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6\": rpc error: code = NotFound desc = could not find container \"bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6\": container with ID starting with bd6fd9509da4f607bd3fb506606831c34e76f53430e14b093a3ac43af84b19b6 not found: ID does not exist"
Nov 03 20:19:13.045013 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:19:13.044982    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17af1c04_f9ef_4e5b_808b_b1c15e9eb5f2.slice/crio-7713a2ea543a81b019d2462f4fcaacd795485208fae91f2ecd6d61768860f13e WatchSource:0}: Error finding container 7713a2ea543a81b019d2462f4fcaacd795485208fae91f2ecd6d61768860f13e: Status 404 returned error can't find the container with id 7713a2ea543a81b019d2462f4fcaacd795485208fae91f2ecd6d61768860f13e
Nov 03 20:19:13.045743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.045616    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"]
Nov 03 20:19:13.048647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.048612    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:13.048758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.048693    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:13.048758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.048740    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:13.104557 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:19:13.104521    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cf104b_b5ba_4ed1_a50d_65f867fc69cb.slice/crio-c0fe6a92cdef72e348974fee24a84da877084a3b6b2671b10b813648f3352a94 WatchSource:0}: Error finding container c0fe6a92cdef72e348974fee24a84da877084a3b6b2671b10b813648f3352a94: Status 404 returned error can't find the container with id c0fe6a92cdef72e348974fee24a84da877084a3b6b2671b10b813648f3352a94
Nov 03 20:19:13.105219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.105176    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"]
Nov 03 20:19:13.394067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.393981    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:13.547184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.547056    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab761e2b-c2a0-4a41-a8ec-efdc9df389e2" path="/var/lib/kubelet/pods/ab761e2b-c2a0-4a41-a8ec-efdc9df389e2/volumes"
Nov 03 20:19:13.799494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.782099    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br" event={"ID":"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb","Type":"ContainerStarted","Data":"7cd3d9a082741581f3f6645eddf50c2fe6f494a0f767f35166bb2a22f7f24d68"}
Nov 03 20:19:13.799494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.782174    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br" event={"ID":"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb","Type":"ContainerStarted","Data":"c0fe6a92cdef72e348974fee24a84da877084a3b6b2671b10b813648f3352a94"}
Nov 03 20:19:13.799494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.787566    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" event={"ID":"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2","Type":"ContainerStarted","Data":"cb2ed9455eb85fb224b7c77c4497ff3239552f8d03ab02b8a012f53958a116d9"}
Nov 03 20:19:13.799494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.789983    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" event={"ID":"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2","Type":"ContainerStarted","Data":"7713a2ea543a81b019d2462f4fcaacd795485208fae91f2ecd6d61768860f13e"}
Nov 03 20:19:13.806247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.804817    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br" podStartSLOduration=1.804799483 podStartE2EDuration="1.804799483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:13.804440048 +0000 UTC m=+2346.473641521" watchObservedRunningTime="2026-11-03 20:19:13.804799483 +0000 UTC m=+2346.474000954"
Nov 03 20:19:13.827120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:13.827067    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podStartSLOduration=9.827053258 podStartE2EDuration="9.827053258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:13.824854653 +0000 UTC m=+2346.494056145" watchObservedRunningTime="2026-11-03 20:19:13.827053258 +0000 UTC m=+2346.496254728"
Nov 03 20:19:14.219294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:14.219264    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:14.361281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:14.352056    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:14.635117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:14.635027    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:14.636443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:14.636417    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 20:19:14.636590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:14.636471    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 20:19:14.811628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:14.811589    5282 generic.go:356] "Generic (PLEG): container finished" podID="c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" containerID="7cd3d9a082741581f3f6645eddf50c2fe6f494a0f767f35166bb2a22f7f24d68" exitCode=0
Nov 03 20:19:14.811824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:14.811668    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br" event={"ID":"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb","Type":"ContainerDied","Data":"7cd3d9a082741581f3f6645eddf50c2fe6f494a0f767f35166bb2a22f7f24d68"}
Nov 03 20:19:15.107306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.107267    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:15.325766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.325718    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"]
Nov 03 20:19:15.331818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.331790    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"]
Nov 03 20:19:15.331947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.331884    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.463281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.463179    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.463447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.463325    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxnv\" (UniqueName: \"kubernetes.io/projected/10273c46-7990-4f1d-8d7b-864cd61ef0b9-kube-api-access-nhxnv\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.463518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.463481    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.565475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.565440    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.565693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.565678    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.565778 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.565767    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxnv\" (UniqueName: \"kubernetes.io/projected/10273c46-7990-4f1d-8d7b-864cd61ef0b9-kube-api-access-nhxnv\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.566301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.566278    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.567794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.567773    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.573131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.573109    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxnv\" (UniqueName: \"kubernetes.io/projected/10273c46-7990-4f1d-8d7b-864cd61ef0b9-kube-api-access-nhxnv\") pod \"maas-api-revoke-sub-0ad4dd4c-r8b7b\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:15.635288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.635259    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 20:19:15.635446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.635309    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 20:19:15.654804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:15.654774    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:16.082996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.073381    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"]
Nov 03 20:19:16.111248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.103290    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:16.118845 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:19:16.118605    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10273c46_7990_4f1d_8d7b_864cd61ef0b9.slice/crio-9769292bb166f0e8fc6150c1a85ca6117f0ad60dfee2fc024e5c0f9bb11da073 WatchSource:0}: Error finding container 9769292bb166f0e8fc6150c1a85ca6117f0ad60dfee2fc024e5c0f9bb11da073: Status 404 returned error can't find the container with id 9769292bb166f0e8fc6150c1a85ca6117f0ad60dfee2fc024e5c0f9bb11da073
Nov 03 20:19:16.318069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.317996    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:16.370802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.370773    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-openshift-ca\") pod \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") "
Nov 03 20:19:16.371302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.370868    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-xks-ca\") pod \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") "
Nov 03 20:19:16.371302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.371034    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-kube-api-access-xwmbm\" (UniqueName: \"kubernetes.io/projected/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-kube-api-access-xwmbm\") pod \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\" (UID: \"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb\") "
Nov 03 20:19:16.371302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.371136    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-openshift-ca" pod "c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" (UID: "c8cf104b-b5ba-4ed1-a50d-65f867fc69cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:16.371757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.371731    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:16.373026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.373003    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-xks-ca" pod "c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" (UID: "c8cf104b-b5ba-4ed1-a50d-65f867fc69cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:16.373176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.373163    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-kube-api-access-xwmbm" pod "c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" (UID: "c8cf104b-b5ba-4ed1-a50d-65f867fc69cb"). InnerVolumeSpecName "kube-api-access-xwmbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:16.473121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.473084    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:16.473121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.473120    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwmbm\" (UniqueName: \"kubernetes.io/projected/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb-kube-api-access-xwmbm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:16.544370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.544340    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:16.647742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.645145    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 20:19:16.647742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.645224    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 20:19:16.831960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.831932    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"
Nov 03 20:19:16.832160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.831965    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br" event={"ID":"c8cf104b-b5ba-4ed1-a50d-65f867fc69cb","Type":"ContainerDied","Data":"c0fe6a92cdef72e348974fee24a84da877084a3b6b2671b10b813648f3352a94"}
Nov 03 20:19:16.832160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.832010    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0fe6a92cdef72e348974fee24a84da877084a3b6b2671b10b813648f3352a94"
Nov 03 20:19:16.833897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.833875    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b" event={"ID":"10273c46-7990-4f1d-8d7b-864cd61ef0b9","Type":"ContainerStarted","Data":"45232dcff46a1f8a028feb3e5aa4a00283603068d93c54bf4d80280c73569ff3"}
Nov 03 20:19:16.833993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.833915    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b" event={"ID":"10273c46-7990-4f1d-8d7b-864cd61ef0b9","Type":"ContainerStarted","Data":"9769292bb166f0e8fc6150c1a85ca6117f0ad60dfee2fc024e5c0f9bb11da073"}
Nov 03 20:19:16.854579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:16.854526    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b" podStartSLOduration=1.8545125599999999 podStartE2EDuration="1.85451256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:16.851061239 +0000 UTC m=+2349.520262711" watchObservedRunningTime="2026-11-03 20:19:16.85451256 +0000 UTC m=+2349.523714030"
Nov 03 20:19:17.669224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:17.655397    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 20:19:17.700792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:17.700695    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 20:19:17.775416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:17.760958    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"]
Nov 03 20:19:17.775416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:17.768730    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:17.844894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:17.844858    5282 generic.go:356] "Generic (PLEG): container finished" podID="10273c46-7990-4f1d-8d7b-864cd61ef0b9" containerID="45232dcff46a1f8a028feb3e5aa4a00283603068d93c54bf4d80280c73569ff3" exitCode=0
Nov 03 20:19:17.845066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:17.844916    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b" event={"ID":"10273c46-7990-4f1d-8d7b-864cd61ef0b9","Type":"ContainerDied","Data":"45232dcff46a1f8a028feb3e5aa4a00283603068d93c54bf4d80280c73569ff3"}
Nov 03 20:19:17.892221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:17.889032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:18.013219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:18.012119    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:18.635559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:18.635520    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 20:19:18.635751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:18.635572    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 20:19:19.103076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.103012    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:19.126925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.126896    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-openshift-ca\") pod \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") "
Nov 03 20:19:19.127069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.127011    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10273c46-7990-4f1d-8d7b-864cd61ef0b9-kube-api-access-nhxnv\" (UniqueName: \"kubernetes.io/projected/10273c46-7990-4f1d-8d7b-864cd61ef0b9-kube-api-access-nhxnv\") pod \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") "
Nov 03 20:19:19.127069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.127047    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-xks-ca\") pod \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\" (UID: \"10273c46-7990-4f1d-8d7b-864cd61ef0b9\") "
Nov 03 20:19:19.127312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.127286    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-openshift-ca" pod "10273c46-7990-4f1d-8d7b-864cd61ef0b9" (UID: "10273c46-7990-4f1d-8d7b-864cd61ef0b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:19.127745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.127725    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-openshift-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:19.129115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.129094    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-xks-ca" pod "10273c46-7990-4f1d-8d7b-864cd61ef0b9" (UID: "10273c46-7990-4f1d-8d7b-864cd61ef0b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:19.129240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.129225    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10273c46-7990-4f1d-8d7b-864cd61ef0b9-kube-api-access-nhxnv" pod "10273c46-7990-4f1d-8d7b-864cd61ef0b9" (UID: "10273c46-7990-4f1d-8d7b-864cd61ef0b9"). InnerVolumeSpecName "kube-api-access-nhxnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:19.240237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.231838    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:19.259223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.253169    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxnv\" (UniqueName: \"kubernetes.io/projected/10273c46-7990-4f1d-8d7b-864cd61ef0b9-kube-api-access-nhxnv\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:19.259223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.253221    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10273c46-7990-4f1d-8d7b-864cd61ef0b9-maas-api-xks-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:19.425959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.417979    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:19.635100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.635059    5282 patch_prober.go:29] interesting pod/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 20:19:19.635290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.635126    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 20:19:19.865878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.865771    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"
Nov 03 20:19:19.866051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.865872    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b" event={"ID":"10273c46-7990-4f1d-8d7b-864cd61ef0b9","Type":"ContainerDied","Data":"9769292bb166f0e8fc6150c1a85ca6117f0ad60dfee2fc024e5c0f9bb11da073"}
Nov 03 20:19:19.866051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:19.865971    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9769292bb166f0e8fc6150c1a85ca6117f0ad60dfee2fc024e5c0f9bb11da073"
Nov 03 20:19:20.191917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:20.191769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:20.641074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:20.641042    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:20.641934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:20.641909    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:20.643366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:20.643343    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:19:20.802080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:20.802042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:21.443835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:21.443802    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:21.568221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:21.563183    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:22.192575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:22.192534    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:22.761847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:22.761807    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx" podUID="2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" containerName="payload-pre-processing" containerID="cri-o://22e4d7c590f4c69a7c4d4907440f5f6bec1b053873947979ae8e76f69ffc7d03" gracePeriod=25
Nov 03 20:19:22.923448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:22.923416    5282 generic.go:356] "Generic (PLEG): container finished" podID="2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" containerID="22e4d7c590f4c69a7c4d4907440f5f6bec1b053873947979ae8e76f69ffc7d03" exitCode=0
Nov 03 20:19:22.923631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:22.923617    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx" event={"ID":"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd","Type":"ContainerDied","Data":"22e4d7c590f4c69a7c4d4907440f5f6bec1b053873947979ae8e76f69ffc7d03"}
Nov 03 20:19:23.098192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.098165    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:19:23.105582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.105535    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:23.176366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.176325    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-kube-api-access-6lfgd\" (UniqueName: \"kubernetes.io/projected/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-kube-api-access-6lfgd\") pod \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") "
Nov 03 20:19:23.187572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.187528    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-plugins-config-volume\") pod \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\" (UID: \"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd\") "
Nov 03 20:19:23.195143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.195103    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-plugins-config-volume" pod "2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" (UID: "2e9e8f3c-7788-40ef-94fd-8586ab4c17dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:23.261501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.248743    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-kube-api-access-6lfgd" pod "2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" (UID: "2e9e8f3c-7788-40ef-94fd-8586ab4c17dd"). InnerVolumeSpecName "kube-api-access-6lfgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:23.261501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.249085    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:23.350709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.350628    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfgd\" (UniqueName: \"kubernetes.io/projected/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd-kube-api-access-6lfgd\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:23.863242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.863180    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"]
Nov 03 20:19:23.863720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.863425    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy" containerID="cri-o://9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c" gracePeriod=30
Nov 03 20:19:23.899977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.899934    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:23.936613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.936582    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"
Nov 03 20:19:23.936811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.936581    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx" event={"ID":"2e9e8f3c-7788-40ef-94fd-8586ab4c17dd","Type":"ContainerDied","Data":"b0bada3412eca1be643dd062e2de5bc42ad2662ddfd1e694108ab801f127afb0"}
Nov 03 20:19:23.936811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.936735    5282 scope.go:122] "RemoveContainer" containerID="22e4d7c590f4c69a7c4d4907440f5f6bec1b053873947979ae8e76f69ffc7d03"
Nov 03 20:19:23.968327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:23.968296    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"]
Nov 03 20:19:24.009227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.002751    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-facc79-5ddbdd97dc-6m6qx"]
Nov 03 20:19:24.139889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.139804    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"]
Nov 03 20:19:24.140069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.140046    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy" containerID="cri-o://fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005" gracePeriod=30
Nov 03 20:19:24.713775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.713740    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"]
Nov 03 20:19:24.714793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.714770    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" containerName="revoke-keys"
Nov 03 20:19:24.714793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.714793    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" containerName="revoke-keys"
Nov 03 20:19:24.714984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.714805    5282 cpu_manager.go:414] "containerMap: removing container" podUID="10273c46-7990-4f1d-8d7b-864cd61ef0b9" containerName="revoke-keys"
Nov 03 20:19:24.714984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.714813    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10273c46-7990-4f1d-8d7b-864cd61ef0b9" containerName="revoke-keys"
Nov 03 20:19:24.714984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.714850    5282 cpu_manager.go:414] "containerMap: removing container" podUID="2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" containerName="payload-pre-processing"
Nov 03 20:19:24.714984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.714858    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" containerName="payload-pre-processing"
Nov 03 20:19:24.715185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.715128    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" containerName="payload-pre-processing"
Nov 03 20:19:24.715185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.715161    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" containerName="revoke-keys"
Nov 03 20:19:24.715185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.715170    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="10273c46-7990-4f1d-8d7b-864cd61ef0b9" containerName="revoke-keys"
Nov 03 20:19:24.721618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.721585    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:24.723554 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.723531    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.728129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.727238    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"]
Nov 03 20:19:24.729946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.729925    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-46c34c\"/\"openshift-service-ca.crt\""
Nov 03 20:19:24.731294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.731272    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-46c34c\"/\"kube-root-ca.crt\""
Nov 03 20:19:24.731442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.731420    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-46c34c\"/\"test-model-46c34c-kserve-self-signed-certs\""
Nov 03 20:19:24.731525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.731501    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-46c34c\"/\"default-dockercfg-9lsx8\""
Nov 03 20:19:24.781678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.781648    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-tmp-dir\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.781876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.781777    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-model-cache\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.781876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.781869    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-home\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.782038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.781907    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-dshm\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.782072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.782036    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b272424b-a840-4c8f-9622-09136854de34-tls-certs\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.782110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.782098    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgcp\" (UniqueName: \"kubernetes.io/projected/b272424b-a840-4c8f-9622-09136854de34-kube-api-access-ldgcp\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.852918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.852889    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.897378    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-tmp-dir\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.897504    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-model-cache\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.897554    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-home\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.897589    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-dshm\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.897681    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b272424b-a840-4c8f-9622-09136854de34-tls-certs\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.897771    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgcp\" (UniqueName: \"kubernetes.io/projected/b272424b-a840-4c8f-9622-09136854de34-kube-api-access-ldgcp\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.898819    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-tmp-dir\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.910221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.899081    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-model-cache\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.920436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.911671    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-home\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.937239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.933270    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgcp\" (UniqueName: \"kubernetes.io/projected/b272424b-a840-4c8f-9622-09136854de34-kube-api-access-ldgcp\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:24.963221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:24.958885    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b272424b-a840-4c8f-9622-09136854de34-tls-certs\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:25.013236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:25.007934    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-dshm\") pod \"test-model-46c34c-kserve-7b557f9477-826dx\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") " pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:25.041218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:25.040676    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:25.459468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:25.459436    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9e8f3c-7788-40ef-94fd-8586ab4c17dd" path="/var/lib/kubelet/pods/2e9e8f3c-7788-40ef-94fd-8586ab4c17dd/volumes"
Nov 03 20:19:25.483405 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:19:25.483376    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb272424b_a840_4c8f_9622_09136854de34.slice/crio-ea96f836e3c0c54d086ed1f08adac49cd4819c1eabdeca3f4160f656ebad211b WatchSource:0}: Error finding container ea96f836e3c0c54d086ed1f08adac49cd4819c1eabdeca3f4160f656ebad211b: Status 404 returned error can't find the container with id ea96f836e3c0c54d086ed1f08adac49cd4819c1eabdeca3f4160f656ebad211b
Nov 03 20:19:25.484440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:25.484417    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"]
Nov 03 20:19:25.763706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:25.763678    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:26.042433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.042393    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx" event={"ID":"b272424b-a840-4c8f-9622-09136854de34","Type":"ContainerStarted","Data":"ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62"}
Nov 03 20:19:26.042843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.042440    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx" event={"ID":"b272424b-a840-4c8f-9622-09136854de34","Type":"ContainerStarted","Data":"ea96f836e3c0c54d086ed1f08adac49cd4819c1eabdeca3f4160f656ebad211b"}
Nov 03 20:19:26.059658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.059604    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx" podStartSLOduration=2.059589484 podStartE2EDuration="2.059589484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:26.057685001 +0000 UTC m=+2358.726886471" watchObservedRunningTime="2026-11-03 20:19:26.059589484 +0000 UTC m=+2358.728791020"
Nov 03 20:19:26.376608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.376530    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"]
Nov 03 20:19:26.376858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.376831    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9" podUID="2662b650-8f0f-4178-9751-d210b42fabed" containerName="maas-api" containerID="cri-o://5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8" gracePeriod=30
Nov 03 20:19:26.460958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.460925    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:26.542361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.542314    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"]
Nov 03 20:19:26.543438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.542806    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c" podUID="ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" containerName="maas-api" containerID="cri-o://51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f" gracePeriod=30
Nov 03 20:19:26.619881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.619852    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:26.701629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.701595    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"]
Nov 03 20:19:26.706159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.706131    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-43b780-bkpnf"]
Nov 03 20:19:26.735366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.735339    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:19:26.741582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.741559    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:26.776314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.776266    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"]
Nov 03 20:19:26.777755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.777727    5282 cpu_manager.go:414] "containerMap: removing container" podUID="2662b650-8f0f-4178-9751-d210b42fabed" containerName="maas-api"
Nov 03 20:19:26.777755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.777754    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2662b650-8f0f-4178-9751-d210b42fabed" containerName="maas-api"
Nov 03 20:19:26.777977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.777961    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2662b650-8f0f-4178-9751-d210b42fabed" containerName="maas-api"
Nov 03 20:19:26.783459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.783433    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.785740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.785717    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"]
Nov 03 20:19:26.788302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.788260    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2c6c50-serving-cert\""
Nov 03 20:19:26.817584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.817551    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-metrics-certs\") pod \"2662b650-8f0f-4178-9751-d210b42fabed\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") "
Nov 03 20:19:26.817722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.817617    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2662b650-8f0f-4178-9751-d210b42fabed-kube-api-access-thklf\" (UniqueName: \"kubernetes.io/projected/2662b650-8f0f-4178-9751-d210b42fabed-kube-api-access-thklf\") pod \"2662b650-8f0f-4178-9751-d210b42fabed\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") "
Nov 03 20:19:26.817863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.817845    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-maas-api-tls\") pod \"2662b650-8f0f-4178-9751-d210b42fabed\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") "
Nov 03 20:19:26.817938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.817896    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2662b650-8f0f-4178-9751-d210b42fabed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2662b650-8f0f-4178-9751-d210b42fabed-openshift-service-ca\") pod \"2662b650-8f0f-4178-9751-d210b42fabed\" (UID: \"2662b650-8f0f-4178-9751-d210b42fabed\") "
Nov 03 20:19:26.818254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.818191    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-metrics-certs\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.818413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.818386    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2662b650-8f0f-4178-9751-d210b42fabed-openshift-service-ca" pod "2662b650-8f0f-4178-9751-d210b42fabed" (UID: "2662b650-8f0f-4178-9751-d210b42fabed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:26.818487 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.818413    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af36f373-ad1c-474c-845e-95586c8304ef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.818681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.818656    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtdf\" (UniqueName: \"kubernetes.io/projected/af36f373-ad1c-474c-845e-95586c8304ef-kube-api-access-qbtdf\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.818770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.818704    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-maas-api-tls\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.819066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.819040    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2662b650-8f0f-4178-9751-d210b42fabed-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:26.821052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.821024    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-metrics-certs" pod "2662b650-8f0f-4178-9751-d210b42fabed" (UID: "2662b650-8f0f-4178-9751-d210b42fabed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:26.821746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.821719    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-maas-api-tls" pod "2662b650-8f0f-4178-9751-d210b42fabed" (UID: "2662b650-8f0f-4178-9751-d210b42fabed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:26.821861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.821837    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2662b650-8f0f-4178-9751-d210b42fabed-kube-api-access-thklf" pod "2662b650-8f0f-4178-9751-d210b42fabed" (UID: "2662b650-8f0f-4178-9751-d210b42fabed"). InnerVolumeSpecName "kube-api-access-thklf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:26.841726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.841700    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.936897    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af36f373-ad1c-474c-845e-95586c8304ef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.937154    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtdf\" (UniqueName: \"kubernetes.io/projected/af36f373-ad1c-474c-845e-95586c8304ef-kube-api-access-qbtdf\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.937192    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-maas-api-tls\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.937484    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-metrics-certs\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.937649    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.937668    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thklf\" (UniqueName: \"kubernetes.io/projected/2662b650-8f0f-4178-9751-d210b42fabed-kube-api-access-thklf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.937685    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2662b650-8f0f-4178-9751-d210b42fabed-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:26.989392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:26.955678    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af36f373-ad1c-474c-845e-95586c8304ef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:27.176708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.066060    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"]
Nov 03 20:19:27.176708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.073864    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-metrics-certs\") pod \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") "
Nov 03 20:19:27.176708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.073932    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-maas-api-tls\") pod \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") "
Nov 03 20:19:27.176708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.158998    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-openshift-service-ca\") pod \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") "
Nov 03 20:19:27.176708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.159299    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-kube-api-access-g5msm\" (UniqueName: \"kubernetes.io/projected/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-kube-api-access-g5msm\") pod \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\" (UID: \"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11\") "
Nov 03 20:19:27.214222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.193538    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-openshift-service-ca" pod "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" (UID: "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:27.262281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.229887    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:27.262281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.230383    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:27.262281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.240576    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" containerName="maas-api"
Nov 03 20:19:27.262281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.240604    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" containerName="maas-api"
Nov 03 20:19:27.262281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.240888    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" containerName="maas-api"
Nov 03 20:19:27.411259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.411224    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-kube-api-access-g5msm" pod "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" (UID: "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11"). InnerVolumeSpecName "kube-api-access-g5msm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:27.414561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.414524    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:27.414856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.414836    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"]
Nov 03 20:19:27.415097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.415076    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.419563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.416994    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-metrics-certs\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:27.419563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.417290    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:27.423001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.422018    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-27d3ad-openshift-default-dockercfg-wt2s4\""
Nov 03 20:19:27.426593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.423932    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-maas-api-tls\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:27.426593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.424569    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtdf\" (UniqueName: \"kubernetes.io/projected/af36f373-ad1c-474c-845e-95586c8304ef-kube-api-access-qbtdf\") pod \"maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:27.430869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.430843    5282 generic.go:356] "Generic (PLEG): container finished" podID="2662b650-8f0f-4178-9751-d210b42fabed" containerID="5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8" exitCode=0
Nov 03 20:19:27.431119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.431097    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9" event={"ID":"2662b650-8f0f-4178-9751-d210b42fabed","Type":"ContainerDied","Data":"5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8"}
Nov 03 20:19:27.431180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.431156    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9" event={"ID":"2662b650-8f0f-4178-9751-d210b42fabed","Type":"ContainerDied","Data":"12360eac9c1def44c8c103e6f211b3e1d144774aeff2448b0471e23b7c86ca65"}
Nov 03 20:19:27.431368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.431351    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"
Nov 03 20:19:27.432401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.432376    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-metrics-certs" pod "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" (UID: "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:27.435101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.431189    5282 scope.go:122] "RemoveContainer" containerID="5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8"
Nov 03 20:19:27.442413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.442381    5282 generic.go:356] "Generic (PLEG): container finished" podID="ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" containerID="51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f" exitCode=0
Nov 03 20:19:27.444753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.444730    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-maas-api-tls" pod "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" (UID: "ecf175a9-bb9b-4a04-abd2-6d386d9e6b11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:27.444945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.444922    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"
Nov 03 20:19:27.444945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.444941    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c" event={"ID":"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11","Type":"ContainerDied","Data":"51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f"}
Nov 03 20:19:27.445070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.444984    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c" event={"ID":"ecf175a9-bb9b-4a04-abd2-6d386d9e6b11","Type":"ContainerDied","Data":"f3f03f08825f2ee7f73eee4fdccff025690d0416d5b37d88065e9d5a690b47a2"}
Nov 03 20:19:27.447974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.447947    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347cb059-324b-4249-b74b-2a18fd8d045a-istiod-ca-cert\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.447992    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-data\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448212    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-credential-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448248    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-istio-token\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448383    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-envoy\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448477    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448518    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-certs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448550    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6t2\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-kube-api-access-lg6t2\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448607    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347cb059-324b-4249-b74b-2a18fd8d045a-istio-podinfo\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448734    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5msm\" (UniqueName: \"kubernetes.io/projected/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-kube-api-access-g5msm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448748    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:27.448831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.448759    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:27.456561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.456543    5282 scope.go:122] "RemoveContainer" containerID="5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8"
Nov 03 20:19:27.456862 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:27.456842    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8\": container with ID starting with 5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8 not found: ID does not exist" containerID="5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8"
Nov 03 20:19:27.456906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.456870    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8"} err="failed to get container status \"5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8\": rpc error: code = NotFound desc = could not find container \"5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8\": container with ID starting with 5a74302f1c9b4bbb1d0d82693f0599e1d2c103f98e493d2779fdf5144f1e84b8 not found: ID does not exist"
Nov 03 20:19:27.456906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.456887    5282 scope.go:122] "RemoveContainer" containerID="51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f"
Nov 03 20:19:27.464832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.464810    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b289275f-e02e-4411-a617-885bfa4cb876" path="/var/lib/kubelet/pods/b289275f-e02e-4411-a617-885bfa4cb876/volumes"
Nov 03 20:19:27.500107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.500024    5282 scope.go:122] "RemoveContainer" containerID="51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f"
Nov 03 20:19:27.502456 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:27.502427    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f\": container with ID starting with 51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f not found: ID does not exist" containerID="51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f"
Nov 03 20:19:27.502586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.502466    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f"} err="failed to get container status \"51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f\": rpc error: code = NotFound desc = could not find container \"51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f\": container with ID starting with 51aa489283cad77067210b2c317c35259d842c1dc9f6f796214c4417d7d16d7f not found: ID does not exist"
Nov 03 20:19:27.508312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.508280    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"]
Nov 03 20:19:27.512344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.512280    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-43b780-68f4df9cc4-wrfx9"]
Nov 03 20:19:27.516784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.516758    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"]
Nov 03 20:19:27.522815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.522796    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-facc79-77bcc4b7d8-ksr5c"]
Nov 03 20:19:27.550619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.550592    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347cb059-324b-4249-b74b-2a18fd8d045a-istiod-ca-cert\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.550733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.550629    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-data\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.550849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.550828    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-credential-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.550887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.550865    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-istio-token\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.550938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.550922    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-envoy\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.550984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.550973    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551107    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-certs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551144    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-credential-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551156    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6t2\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-kube-api-access-lg6t2\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551119    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-data\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551257    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347cb059-324b-4249-b74b-2a18fd8d045a-istio-podinfo\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551316    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-socket\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551402    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-certs\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.551672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.551459    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347cb059-324b-4249-b74b-2a18fd8d045a-istiod-ca-cert\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.553282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.553262    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-envoy\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.553626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.553608    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347cb059-324b-4249-b74b-2a18fd8d045a-istio-podinfo\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.562117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.562093    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-istio-token\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.562328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.562311    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6t2\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-kube-api-access-lg6t2\") pod \"e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") " pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:27.564420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.564406    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"]
Nov 03 20:19:27.568942 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.568921    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-facc79-24zjq"]
Nov 03 20:19:27.603056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.603032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:27.729519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.729085    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:27.730923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.730445    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:27.784713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:27.784621    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:28.029414 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:19:28.029384    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf36f373_ad1c_474c_845e_95586c8304ef.slice/crio-23072a98702d34834bd8f85160aa24fa0ee847e1cb325e0fa0c874d8e2e23138 WatchSource:0}: Error finding container 23072a98702d34834bd8f85160aa24fa0ee847e1cb325e0fa0c874d8e2e23138: Status 404 returned error can't find the container with id 23072a98702d34834bd8f85160aa24fa0ee847e1cb325e0fa0c874d8e2e23138
Nov 03 20:19:28.031009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.030989    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:19:28.031440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.031406    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"]
Nov 03 20:19:28.088639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.088559    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:28.114643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.113165    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:28.114643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.113333    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:28.114643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.113425    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:28.125171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.125142    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"]
Nov 03 20:19:28.455257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.455129    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl" event={"ID":"af36f373-ad1c-474c-845e-95586c8304ef","Type":"ContainerStarted","Data":"23072a98702d34834bd8f85160aa24fa0ee847e1cb325e0fa0c874d8e2e23138"}
Nov 03 20:19:28.458090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.458063    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" event={"ID":"347cb059-324b-4249-b74b-2a18fd8d045a","Type":"ContainerStarted","Data":"5bcedfd738e5b4906ce66fa796b322c0b81482e249ec88e229de9c87bca2b1ec"}
Nov 03 20:19:28.458267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.458106    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" event={"ID":"347cb059-324b-4249-b74b-2a18fd8d045a","Type":"ContainerStarted","Data":"5607d30570879492306a97b565a5a174f08af8a8f238a65ae882d81ee6168835"}
Nov 03 20:19:28.484772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.484725    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podStartSLOduration=249.484711343 podStartE2EDuration="4m9.484711343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:28.478965216 +0000 UTC m=+2361.148166687" watchObservedRunningTime="2026-11-03 20:19:28.484711343 +0000 UTC m=+2361.153912812"
Nov 03 20:19:28.776506 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.776115    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:28.785152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:28.785123    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:29.157923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.157898    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:19:29.249429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.249395    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-credential-socket\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.249613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.249596    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aff8592-b84b-4bd4-8806-c748cf120d9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aff8592-b84b-4bd4-8806-c748cf120d9e-istiod-ca-cert\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.249702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.249682    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-credential-socket" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.249770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.249690    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-podinfo\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.249770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.249743    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-token\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-token\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.249873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.249829    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-socket\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.250097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250076    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-certs\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.250270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250121    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-kube-api-access-np2cs\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-kube-api-access-np2cs\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.250270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250153    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-data\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.250270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250154    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-socket" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.250270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250166    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aff8592-b84b-4bd4-8806-c748cf120d9e-istiod-ca-cert" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:29.250270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250259    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-envoy\") pod \"8aff8592-b84b-4bd4-8806-c748cf120d9e\" (UID: \"8aff8592-b84b-4bd4-8806-c748cf120d9e\") "
Nov 03 20:19:29.250545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250442    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-certs" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.250545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.250480    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-data" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.251368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.251340    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.251462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.251372    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aff8592-b84b-4bd4-8806-c748cf120d9e-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.251462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.251389    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.251462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.251402    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.251462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.251417    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.252516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.252487    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-podinfo" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:29.252605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.252532    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-kube-api-access-np2cs" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "kube-api-access-np2cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.253134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.253104    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-token" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.253277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.253157    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-envoy" pod "8aff8592-b84b-4bd4-8806-c748cf120d9e" (UID: "8aff8592-b84b-4bd4-8806-c748cf120d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.359320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.359283    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.359482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.359322    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.359482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.359340    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np2cs\" (UniqueName: \"kubernetes.io/projected/8aff8592-b84b-4bd4-8806-c748cf120d9e-kube-api-access-np2cs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.359482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.359356    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aff8592-b84b-4bd4-8806-c748cf120d9e-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.434017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.433993    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:19:29.461236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.460921    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2662b650-8f0f-4178-9751-d210b42fabed" path="/var/lib/kubelet/pods/2662b650-8f0f-4178-9751-d210b42fabed/volumes"
Nov 03 20:19:29.461707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.461594    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caea382a-8001-41e8-b21a-a5d4dbf3eeda" path="/var/lib/kubelet/pods/caea382a-8001-41e8-b21a-a5d4dbf3eeda/volumes"
Nov 03 20:19:29.462258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.462236    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf175a9-bb9b-4a04-abd2-6d386d9e6b11" path="/var/lib/kubelet/pods/ecf175a9-bb9b-4a04-abd2-6d386d9e6b11/volumes"
Nov 03 20:19:29.471155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.471130    5282 generic.go:356] "Generic (PLEG): container finished" podID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerID="9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c" exitCode=0
Nov 03 20:19:29.471327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.471171    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"
Nov 03 20:19:29.471327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.471228    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" event={"ID":"8aff8592-b84b-4bd4-8806-c748cf120d9e","Type":"ContainerDied","Data":"9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c"}
Nov 03 20:19:29.471327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.471277    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l" event={"ID":"8aff8592-b84b-4bd4-8806-c748cf120d9e","Type":"ContainerDied","Data":"967646f2a874323921ed9d64eed48478127fab06bdb3602613cdbfb9af29e1e2"}
Nov 03 20:19:29.471327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.471298    5282 scope.go:122] "RemoveContainer" containerID="9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c"
Nov 03 20:19:29.474250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.474173    5282 generic.go:356] "Generic (PLEG): container finished" podID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerID="fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005" exitCode=0
Nov 03 20:19:29.474250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.474227    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"
Nov 03 20:19:29.474422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.474255    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" event={"ID":"33a96be7-05f9-4135-be1d-9ab1bb17948a","Type":"ContainerDied","Data":"fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005"}
Nov 03 20:19:29.474422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.474399    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp" event={"ID":"33a96be7-05f9-4135-be1d-9ab1bb17948a","Type":"ContainerDied","Data":"237f965007ff9f2097955ea8433a7c7841ce2965ec178ae386cf683ffddf6888"}
Nov 03 20:19:29.490256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.489669    5282 scope.go:122] "RemoveContainer" containerID="9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c"
Nov 03 20:19:29.490256 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:29.490169    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c\": container with ID starting with 9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c not found: ID does not exist" containerID="9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c"
Nov 03 20:19:29.490256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.490217    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c"} err="failed to get container status \"9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c\": rpc error: code = NotFound desc = could not find container \"9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c\": container with ID starting with 9b95525a802ca22d5f22e9981e1b578b807e6b99b1019365fc124c0091ab163c not found: ID does not exist"
Nov 03 20:19:29.490256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.490243    5282 scope.go:122] "RemoveContainer" containerID="fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005"
Nov 03 20:19:29.491441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.491285    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl" event={"ID":"af36f373-ad1c-474c-845e-95586c8304ef","Type":"ContainerStarted","Data":"d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a"}
Nov 03 20:19:29.491441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.491405    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:29.506466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.506443    5282 scope.go:122] "RemoveContainer" containerID="fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005"
Nov 03 20:19:29.506825 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:29.506799    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005\": container with ID starting with fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005 not found: ID does not exist" containerID="fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005"
Nov 03 20:19:29.506906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.506836    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005"} err="failed to get container status \"fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005\": rpc error: code = NotFound desc = could not find container \"fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005\": container with ID starting with fd88c065a9d83d1ff7eeb205553c7cdb815256c5edd096b485243a0f69d91005 not found: ID does not exist"
Nov 03 20:19:29.511854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.511804    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl" podStartSLOduration=295.07301704 podStartE2EDuration="4m55.511791192s" totalImagesPullingTime="438.774152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:29.509113425 +0000 UTC m=+2362.178314895" watchObservedRunningTime="2026-11-03 20:19:29.511791192 +0000 UTC m=+2362.180992662"
Nov 03 20:19:29.527673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.527645    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"]
Nov 03 20:19:29.532412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.532386    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-96x6l"]
Nov 03 20:19:29.563370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563339    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-credential-socket\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.563535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563432    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-token\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-token\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.563535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563503    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-kube-api-access-l7tb7\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-kube-api-access-l7tb7\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.563707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563660    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-credential-socket" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.563764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563705    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-socket\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.563764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563748    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-envoy\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.563888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563864    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a96be7-05f9-4135-be1d-9ab1bb17948a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a96be7-05f9-4135-be1d-9ab1bb17948a-istiod-ca-cert\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.563944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563923    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-podinfo\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.564011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.563972    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-socket" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.564063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.564016    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-certs\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.564155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.564116    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-data\") pod \"33a96be7-05f9-4135-be1d-9ab1bb17948a\" (UID: \"33a96be7-05f9-4135-be1d-9ab1bb17948a\") "
Nov 03 20:19:29.564313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.564291    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a96be7-05f9-4135-be1d-9ab1bb17948a-istiod-ca-cert" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:29.564429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.564325    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-certs" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.564594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.564545    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-data" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.565508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.565469    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.565508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.565500    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33a96be7-05f9-4135-be1d-9ab1bb17948a-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.565737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.565516    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.565737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.565531    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.565737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.565546    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.566178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.566154    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-kube-api-access-l7tb7" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "kube-api-access-l7tb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.566343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.566284    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-envoy" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:29.566343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.566311    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-token" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.566452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.566385    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-podinfo" pod "33a96be7-05f9-4135-be1d-9ab1bb17948a" (UID: "33a96be7-05f9-4135-be1d-9ab1bb17948a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:29.672285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.671192    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.672285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.671248    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tb7\" (UniqueName: \"kubernetes.io/projected/33a96be7-05f9-4135-be1d-9ab1bb17948a-kube-api-access-l7tb7\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.672285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.671264    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.672285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.671277    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33a96be7-05f9-4135-be1d-9ab1bb17948a-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.792884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.792819    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:29.815995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.815954    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"]
Nov 03 20:19:29.834475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.832900    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-sbkvp"]
Nov 03 20:19:29.915296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:29.915260    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:30.289928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:30.289895    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:30.503354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:30.503326    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:30.504144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:30.504126    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:19:30.979998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:30.979965    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:31.124719 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:31.124687    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:31.479308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:31.479103    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" path="/var/lib/kubelet/pods/33a96be7-05f9-4135-be1d-9ab1bb17948a/volumes"
Nov 03 20:19:31.481065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:31.481037    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:31.491236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:31.481659    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" path="/var/lib/kubelet/pods/8aff8592-b84b-4bd4-8806-c748cf120d9e/volumes"
Nov 03 20:19:31.619103 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:31.619070    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:19:32.297799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:32.297766    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:32.494221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:32.490005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:33.512251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:33.512187    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:34.196584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:34.196537    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"]
Nov 03 20:19:34.196901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:34.196869    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy" containerID="cri-o://aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16" gracePeriod=30
Nov 03 20:19:34.377931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:34.377895    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:34.781771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:34.781735    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"]
Nov 03 20:19:35.054272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:35.053364    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:35.603220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:35.589793    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:35.762398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:35.750272    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:36.019177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:35.969227    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:19:36.019177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:35.974557    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:36.082404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.082373    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"]
Nov 03 20:19:36.082741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.082724    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:36.083531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.083491    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy" containerID="cri-o://730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c" gracePeriod=30
Nov 03 20:19:36.168232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.165496    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:36.252594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.252546    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:36.416769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.416682    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:19:36.417296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.416835    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:19:36.579045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.579014    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:19:36.648769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.648718    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" podUID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.151:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:36.656482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:36.656355    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:37.396026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:37.395990    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:37.512758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:37.512726    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:37.795179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:37.795144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:38.596003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:38.595967    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:38.890912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:38.890830    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"]
Nov 03 20:19:38.891115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:38.891094    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy" containerID="cri-o://3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e" gracePeriod=30
Nov 03 20:19:39.160601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.160517    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"]
Nov 03 20:19:39.160752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.160735    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" containerID="cri-o://c693ed9062c20f06a83ff418e7f17a4cd0baeb19351551576852c47f7047b6c7" gracePeriod=30
Nov 03 20:19:39.491974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.491944    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"]
Nov 03 20:19:39.492177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.492152    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy" containerID="cri-o://cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209" gracePeriod=30
Nov 03 20:19:39.512557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.512528    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:19:39.627589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.627560    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-certs\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.627742    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-data\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.627768    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-kube-api-access-nckwx\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-kube-api-access-nckwx\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.627812    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-credential-socket\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.627878    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-certs" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:39.628065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.627938    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8918da4-955f-455e-bd6d-a5f7d58a2867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8918da4-955f-455e-bd6d-a5f7d58a2867-istiod-ca-cert\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628054    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-data" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:39.628439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628081    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-podinfo\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628112    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-envoy\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628138    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-credential-socket" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:39.628439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628147    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-socket\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628243    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-token\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-token\") pod \"a8918da4-955f-455e-bd6d-a5f7d58a2867\" (UID: \"a8918da4-955f-455e-bd6d-a5f7d58a2867\") "
Nov 03 20:19:39.628439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628419    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8918da4-955f-455e-bd6d-a5f7d58a2867-istiod-ca-cert" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:39.628720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.628656    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-socket" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:39.629385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.629279    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.629385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.629305    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.629385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.629321    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.629385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.629334    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8918da4-955f-455e-bd6d-a5f7d58a2867-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.629385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.629350    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.631418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.631396    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-podinfo" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:39.631579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.631453    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-envoy" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:39.631579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.631484    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-kube-api-access-nckwx" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "kube-api-access-nckwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:39.631801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.631781    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:39.631948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.631929    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-token" pod "a8918da4-955f-455e-bd6d-a5f7d58a2867" (UID: "a8918da4-955f-455e-bd6d-a5f7d58a2867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:39.769289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.755402    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.769821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.769782    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.770093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.770078    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:39.770211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:39.770179    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nckwx\" (UniqueName: \"kubernetes.io/projected/a8918da4-955f-455e-bd6d-a5f7d58a2867-kube-api-access-nckwx\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:40.133039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.132958    5282 generic.go:356] "Generic (PLEG): container finished" podID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerID="aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16" exitCode=0
Nov 03 20:19:40.133039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.133009    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"
Nov 03 20:19:40.133329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.133047    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" event={"ID":"a8918da4-955f-455e-bd6d-a5f7d58a2867","Type":"ContainerDied","Data":"aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16"}
Nov 03 20:19:40.133329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.133113    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn" event={"ID":"a8918da4-955f-455e-bd6d-a5f7d58a2867","Type":"ContainerDied","Data":"ad488acda0beb76e0b282cd1563142bedce88153dfc21c88e7cd93f191af19d6"}
Nov 03 20:19:40.133329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.133142    5282 scope.go:122] "RemoveContainer" containerID="aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16"
Nov 03 20:19:40.145670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.145654    5282 scope.go:122] "RemoveContainer" containerID="aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16"
Nov 03 20:19:40.145965 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:40.145947    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16\": container with ID starting with aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16 not found: ID does not exist" containerID="aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16"
Nov 03 20:19:40.146013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.145972    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16"} err="failed to get container status \"aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16\": rpc error: code = NotFound desc = could not find container \"aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16\": container with ID starting with aa3dec484a6db6b98316026d44d1f0e70886b94b3f8f0d17dc663f65a767ea16 not found: ID does not exist"
Nov 03 20:19:40.159802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.159780    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"]
Nov 03 20:19:40.164936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.164912    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-v45zn"]
Nov 03 20:19:40.398631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.398544    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:40.586779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.586741    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"]
Nov 03 20:19:40.588108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.588080    5282 cpu_manager.go:414] "containerMap: removing container" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy"
Nov 03 20:19:40.588108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.588106    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy"
Nov 03 20:19:40.588340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.588241    5282 cpu_manager.go:414] "containerMap: removing container" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy"
Nov 03 20:19:40.589429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.589403    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy"
Nov 03 20:19:40.589581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.589532    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy"
Nov 03 20:19:40.589581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.589545    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy"
Nov 03 20:19:40.589832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.589811    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a96be7-05f9-4135-be1d-9ab1bb17948a" containerName="istio-proxy"
Nov 03 20:19:40.589921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.589843    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" containerName="istio-proxy"
Nov 03 20:19:40.589921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.589861    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aff8592-b84b-4bd4-8806-c748cf120d9e" containerName="istio-proxy"
Nov 03 20:19:40.597042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.597018    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.599102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.599076    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"]
Nov 03 20:19:40.602752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.602730    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0a7343\"/\"default-dockercfg-k6nfr\""
Nov 03 20:19:40.604736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.603630    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0a7343\"/\"openshift-service-ca.crt\""
Nov 03 20:19:40.604736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.603818    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0a7343\"/\"kube-root-ca.crt\""
Nov 03 20:19:40.622114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.622087    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0a7343\"/\"test-model-0a7343-kserve-self-signed-certs\""
Nov 03 20:19:40.658151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.656861    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:40.705988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.705949    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-model-cache\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.706313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.706287    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-home\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.706561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.706537    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tls-certs\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.706757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.706674    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-dshm\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.706757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.706741    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfr9j\" (UniqueName: \"kubernetes.io/projected/2ea21d06-e50b-47f7-9921-fe38f1c010d9-kube-api-access-qfr9j\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.706907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.706825    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tmp-dir\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.901219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.897061    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-dshm\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.901219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.897138    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfr9j\" (UniqueName: \"kubernetes.io/projected/2ea21d06-e50b-47f7-9921-fe38f1c010d9-kube-api-access-qfr9j\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.901219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.897247    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tmp-dir\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.901219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.897447    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-model-cache\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.901219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.897535    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-home\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.901219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.897662    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tls-certs\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.928185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.928084    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-model-cache\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.956991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.933388    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-home\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.960513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.960471    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tmp-dir\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:40.988510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:40.967674    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfr9j\" (UniqueName: \"kubernetes.io/projected/2ea21d06-e50b-47f7-9921-fe38f1c010d9-kube-api-access-qfr9j\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:41.024657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.024624    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tls-certs\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:41.060528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.059716    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-dshm\") pod \"test-model-0a7343-kserve-9df449995-c6d7p\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") " pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:41.255850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.255817    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:41.321228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.320752    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:41.474597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.474565    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8918da4-955f-455e-bd6d-a5f7d58a2867" path="/var/lib/kubelet/pods/a8918da4-955f-455e-bd6d-a5f7d58a2867/volumes"
Nov 03 20:19:41.480868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.480842    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:19:41.571969 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:19:41.571935    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea21d06_e50b_47f7_9921_fe38f1c010d9.slice/crio-8c0cc774a36af7a3b27a77b02561bc105ce0420978f789b2414a2c503f5599a9 WatchSource:0}: Error finding container 8c0cc774a36af7a3b27a77b02561bc105ce0420978f789b2414a2c503f5599a9: Status 404 returned error can't find the container with id 8c0cc774a36af7a3b27a77b02561bc105ce0420978f789b2414a2c503f5599a9
Nov 03 20:19:41.572978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.572897    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"]
Nov 03 20:19:41.653997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.653972    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-envoy\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654044    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-podinfo\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654218    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-credential-socket\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654309    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-kube-api-access-v97x5\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-kube-api-access-v97x5\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654379    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-data\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654488    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-certs\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654554 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654526    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-token\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-token\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654554    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-credential-socket" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:41.654614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654572    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-socket\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654706    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-data" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:41.654752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654747    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f42d2e4-8389-474b-9c77-eb1fd340edad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42d2e4-8389-474b-9c77-eb1fd340edad-istiod-ca-cert\") pod \"0f42d2e4-8389-474b-9c77-eb1fd340edad\" (UID: \"0f42d2e4-8389-474b-9c77-eb1fd340edad\") "
Nov 03 20:19:41.654915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654766    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-certs" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:41.655015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.654982    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-socket" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:41.655309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.655183    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f42d2e4-8389-474b-9c77-eb1fd340edad-istiod-ca-cert" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:41.655642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.655618    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.655746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.655648    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.655746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.655663    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.655746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.655675    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.655746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.655692    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f42d2e4-8389-474b-9c77-eb1fd340edad-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.656601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.656567    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-envoy" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:41.656601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.656581    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-kube-api-access-v97x5" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "kube-api-access-v97x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:41.657050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.657034    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-podinfo" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:41.657116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.657064    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-token" pod "0f42d2e4-8389-474b-9c77-eb1fd340edad" (UID: "0f42d2e4-8389-474b-9c77-eb1fd340edad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:41.757759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.757728    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.757759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.757756    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.758162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.757767    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97x5\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-kube-api-access-v97x5\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.758162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.757778    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f42d2e4-8389-474b-9c77-eb1fd340edad-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:41.893933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:41.893897    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:19:42.390111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.390061    5282 generic.go:356] "Generic (PLEG): container finished" podID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerID="730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c" exitCode=0
Nov 03 20:19:42.390827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.390785    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" event={"ID":"0f42d2e4-8389-474b-9c77-eb1fd340edad","Type":"ContainerDied","Data":"730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c"}
Nov 03 20:19:42.391138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.391055    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh" event={"ID":"0f42d2e4-8389-474b-9c77-eb1fd340edad","Type":"ContainerDied","Data":"ebca3f4007f54d340897fe6f339659d7e746377e912802e71956366495f7ec52"}
Nov 03 20:19:42.391349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.391334    5282 scope.go:122] "RemoveContainer" containerID="730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c"
Nov 03 20:19:42.394619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.392357    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"
Nov 03 20:19:42.407047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.407020    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p" event={"ID":"2ea21d06-e50b-47f7-9921-fe38f1c010d9","Type":"ContainerStarted","Data":"9954b669d010d457ee5f0b91791039ce8ec5e8032e2e71e3192357f31ed50091"}
Nov 03 20:19:42.407254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.407233    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p" event={"ID":"2ea21d06-e50b-47f7-9921-fe38f1c010d9","Type":"ContainerStarted","Data":"8c0cc774a36af7a3b27a77b02561bc105ce0420978f789b2414a2c503f5599a9"}
Nov 03 20:19:42.442792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.441988    5282 scope.go:122] "RemoveContainer" containerID="d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38"
Nov 03 20:19:42.443054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.442997    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p" podStartSLOduration=2.442980204 podStartE2EDuration="2.442980204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:42.442450067 +0000 UTC m=+2375.111651539" watchObservedRunningTime="2026-11-03 20:19:42.442980204 +0000 UTC m=+2375.112181678"
Nov 03 20:19:42.469627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.469587    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"]
Nov 03 20:19:42.469774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.469754    5282 scope.go:122] "RemoveContainer" containerID="730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c"
Nov 03 20:19:42.470945 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:42.470607    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c\": container with ID starting with 730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c not found: ID does not exist" containerID="730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c"
Nov 03 20:19:42.470945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.470649    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c"} err="failed to get container status \"730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c\": rpc error: code = NotFound desc = could not find container \"730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c\": container with ID starting with 730a89aedf51d0202eecfdf65b341ab6418e2a990a6844aa5a56b1a11be40c7c not found: ID does not exist"
Nov 03 20:19:42.470945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.470671    5282 scope.go:122] "RemoveContainer" containerID="d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38"
Nov 03 20:19:42.471869 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:42.471832    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38\": container with ID starting with d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38 not found: ID does not exist" containerID="d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38"
Nov 03 20:19:42.472012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.471880    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38"} err="failed to get container status \"d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38\": rpc error: code = NotFound desc = could not find container \"d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38\": container with ID starting with d9d1bd1989cd2dee0d96680fe23ab741aa0127a8e5e1be8f5332d7f04948cb38 not found: ID does not exist"
Nov 03 20:19:42.473571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.473552    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:42.476733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:42.476709    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-facc79-openshift-default-5bdf4749c9-2l2wh"]
Nov 03 20:19:43.254834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:43.254800    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:43.279238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:43.272310    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:43.325260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:43.321376    5282 patch_prober.go:29] interesting pod/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:19:43.325260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:43.321444    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:19:43.457695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:43.457666    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" path="/var/lib/kubelet/pods/0f42d2e4-8389-474b-9c77-eb1fd340edad/volumes"
Nov 03 20:19:44.362915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.362677    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:19:44.447053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.447011    5282 generic.go:356] "Generic (PLEG): container finished" podID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerID="c693ed9062c20f06a83ff418e7f17a4cd0baeb19351551576852c47f7047b6c7" exitCode=0
Nov 03 20:19:44.448082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.447386    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" event={"ID":"3eb363fe-2c71-4a71-8a21-9b366a5d42a5","Type":"ContainerDied","Data":"c693ed9062c20f06a83ff418e7f17a4cd0baeb19351551576852c47f7047b6c7"}
Nov 03 20:19:44.448082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.447599    5282 scope.go:122] "RemoveContainer" containerID="a0ad6b5004ad2738b17114df3ece7175f71aca0bcd67cbccfc5e3fee6c4a5f2d"
Nov 03 20:19:44.451069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.451033    5282 generic.go:356] "Generic (PLEG): container finished" podID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerID="3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e" exitCode=0
Nov 03 20:19:44.451221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.451092    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"
Nov 03 20:19:44.451221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.451138    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" event={"ID":"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc","Type":"ContainerDied","Data":"3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e"}
Nov 03 20:19:44.451221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.451179    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb" event={"ID":"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc","Type":"ContainerDied","Data":"6ee709430472e7662db34382f38e35657876cecfe606c5118c281f0d3c2a1573"}
Nov 03 20:19:44.472204 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.472173    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:44.482142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.482121    5282 scope.go:122] "RemoveContainer" containerID="3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e"
Nov 03 20:19:44.482756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.482737    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-socket\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.482846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.482814    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-podinfo\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.482910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.482846    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-data\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.482971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.482912    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-certs\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.482971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.482956    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istiod-ca-cert\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.483072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483054    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-envoy\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.483173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483153    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-kube-api-access-6sspj\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-kube-api-access-6sspj\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.483291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483254    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-data" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.483291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483255    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-certs" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.483412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483381    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-socket" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.483533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483432    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-credential-socket\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.483533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483506    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istiod-ca-cert" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:44.483650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483536    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-token\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-token\") pod \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\" (UID: \"1cd7bc8d-1d27-4642-a1fb-676c08dae7bc\") "
Nov 03 20:19:44.483917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.483897    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-credential-socket" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.484346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.484326    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.484443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.484348    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.484443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.484360    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.484443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.484374    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.484443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.484386    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.484897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.484882    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:19:44.485886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.485869    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-token" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:44.486040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.486024    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-podinfo" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:44.486151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.486130    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-kube-api-access-6sspj" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "kube-api-access-6sspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:44.487553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.487534    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-envoy" pod "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" (UID: "1cd7bc8d-1d27-4642-a1fb-676c08dae7bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.495824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.495801    5282 scope.go:122] "RemoveContainer" containerID="3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e"
Nov 03 20:19:44.496238 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:44.496176    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e\": container with ID starting with 3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e not found: ID does not exist" containerID="3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e"
Nov 03 20:19:44.496493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.496396    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e"} err="failed to get container status \"3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e\": rpc error: code = NotFound desc = could not find container \"3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e\": container with ID starting with 3a6f043af9a36f216e8acd8ad1fab59357644c8136c1a2a3d5accd2b61e2111e not found: ID does not exist"
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.585338    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istiod-ca-cert\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.585407    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-kube-api-access-c5wx7\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-kube-api-access-c5wx7\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.585472    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-data\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.585642    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-podinfo\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.585695    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-envoy\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.585790    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-token\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-token\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.585878    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-socket\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.586018    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-certs\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.586064    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-credential-socket\") pod \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\" (UID: \"3eb363fe-2c71-4a71-8a21-9b366a5d42a5\") "
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.586877    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.586902    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.586918    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sspj\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-kube-api-access-6sspj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.586931    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.588979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.588624    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istiod-ca-cert" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:44.591703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.591455    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-certs" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.591703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.591660    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-data" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.592471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.591944    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-credential-socket" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.594233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.594211    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-socket" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.599076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.599040    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-kube-api-access-c5wx7" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "kube-api-access-c5wx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:44.606654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.606608    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-podinfo" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:44.608108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.608069    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-envoy" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.608410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.608379    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-token" pod "3eb363fe-2c71-4a71-8a21-9b366a5d42a5" (UID: "3eb363fe-2c71-4a71-8a21-9b366a5d42a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689470    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689503    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689520    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689534    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689547    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689563    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689577    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689591    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.691262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.689605    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5wx7\" (UniqueName: \"kubernetes.io/projected/3eb363fe-2c71-4a71-8a21-9b366a5d42a5-kube-api-access-c5wx7\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:44.804048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.804012    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"]
Nov 03 20:19:44.808951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.808919    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-978hb"]
Nov 03 20:19:44.869081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.868452    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:19:44.876015 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:44.872576    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd7bc8d_1d27_4642_a1fb_676c08dae7bc.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 20:19:44.920663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.920622    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-data\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.921064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.921027    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-kube-api-access-4b8rf\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-kube-api-access-4b8rf\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.921228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.921212    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-token\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-token\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.921373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.921356    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-podinfo\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.921564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.921549    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-socket\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.921670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.921657    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-certs\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.921880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.921864    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istiod-ca-cert\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.922014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.921999    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-envoy\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.922118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.922102    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-credential-socket\") pod \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\" (UID: \"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9\") "
Nov 03 20:19:44.928935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.928902    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-certs" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.929381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.929358    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-socket" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.929491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.929450    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istiod-ca-cert" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:44.930005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.929977    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-data" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.931378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.931337    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-credential-socket" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.931777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.931736    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-envoy" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:44.932634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.932601    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-kube-api-access-4b8rf" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "kube-api-access-4b8rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:44.933780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.933743    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-podinfo" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:44.934093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:44.934070    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-token" pod "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" (UID: "a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:45.025621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025586    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025767    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025789    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025805    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025818    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025832    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025846    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8rf\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-kube-api-access-4b8rf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025862    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.025959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.025876    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:45.303822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.303724    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:45.459067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.459027    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" path="/var/lib/kubelet/pods/1cd7bc8d-1d27-4642-a1fb-676c08dae7bc/volumes"
Nov 03 20:19:45.461542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.461512    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"
Nov 03 20:19:45.461673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.461543    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m" event={"ID":"3eb363fe-2c71-4a71-8a21-9b366a5d42a5","Type":"ContainerDied","Data":"70531b58fba1f1a1f8f96dc217d6f43dda0848bf4044da6317540f9e91036ce7"}
Nov 03 20:19:45.461673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.461596    5282 scope.go:122] "RemoveContainer" containerID="c693ed9062c20f06a83ff418e7f17a4cd0baeb19351551576852c47f7047b6c7"
Nov 03 20:19:45.466561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.466533    5282 generic.go:356] "Generic (PLEG): container finished" podID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerID="cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209" exitCode=0
Nov 03 20:19:45.466690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.466673    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" event={"ID":"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9","Type":"ContainerDied","Data":"cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209"}
Nov 03 20:19:45.466742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.466724    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8" event={"ID":"a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9","Type":"ContainerDied","Data":"b798d6e73f022e461ca03a8555dff5c823fa58cf9a5a733ba08a8fd6e543465c"}
Nov 03 20:19:45.466811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.466798    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"
Nov 03 20:19:45.494323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.482000    5282 scope.go:122] "RemoveContainer" containerID="cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209"
Nov 03 20:19:45.519703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.519675    5282 scope.go:122] "RemoveContainer" containerID="e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d"
Nov 03 20:19:45.525859 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.525832    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"]
Nov 03 20:19:45.531347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.531296    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-vj76m"]
Nov 03 20:19:45.531431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.531369    5282 scope.go:122] "RemoveContainer" containerID="cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209"
Nov 03 20:19:45.531702 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:45.531683    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209\": container with ID starting with cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209 not found: ID does not exist" containerID="cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209"
Nov 03 20:19:45.531741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.531710    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209"} err="failed to get container status \"cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209\": rpc error: code = NotFound desc = could not find container \"cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209\": container with ID starting with cab86e41e345f149800474fa6ea38ea7fac32382f864a22c4f177ee9efb98209 not found: ID does not exist"
Nov 03 20:19:45.531741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.531726    5282 scope.go:122] "RemoveContainer" containerID="e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d"
Nov 03 20:19:45.531977 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:45.531956    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d\": container with ID starting with e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d not found: ID does not exist" containerID="e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d"
Nov 03 20:19:45.532041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.531989    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d"} err="failed to get container status \"e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d\": rpc error: code = NotFound desc = could not find container \"e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d\": container with ID starting with e1c23ee5cce67f2f56029bba67bd35992ea001051ac1ca230e52fdf1aea4334d not found: ID does not exist"
Nov 03 20:19:45.543591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.543562    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"]
Nov 03 20:19:45.549044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.549024    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-57zg8"]
Nov 03 20:19:45.995714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:45.995679    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:46.324618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:46.324528    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"]
Nov 03 20:19:46.426319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:46.426271    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:46.534645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:46.534605    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"]
Nov 03 20:19:46.570252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:46.570187    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:46.699620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:46.692506    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:46.806814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:46.806775    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:47.529481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:47.529446    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" path="/var/lib/kubelet/pods/3eb363fe-2c71-4a71-8a21-9b366a5d42a5/volumes"
Nov 03 20:19:47.530153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:47.530129    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" path="/var/lib/kubelet/pods/a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9/volumes"
Nov 03 20:19:47.784300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:47.783768    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:48.528338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:48.528304    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:48.641326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:48.641293    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:49.499532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:49.499499    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:49.672591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:49.672555    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:49.921017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:49.920824    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" podUID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" containerName="main" containerID="cri-o://27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375" gracePeriod=45
Nov 03 20:19:50.338896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.338866    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:19:50.403181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403130    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-model-cache\") pod \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") "
Nov 03 20:19:50.403348 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403223    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-home\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-home\") pod \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") "
Nov 03 20:19:50.403433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403412    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tmp-dir\") pod \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") "
Nov 03 20:19:50.403507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403490    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-kube-api-access-wtlnb\" (UniqueName: \"kubernetes.io/projected/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-kube-api-access-wtlnb\") pod \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") "
Nov 03 20:19:50.403624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403608    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-dshm\") pod \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") "
Nov 03 20:19:50.403706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403691    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tls-certs\") pod \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\" (UID: \"3b558ab4-c66b-48b8-a93d-c8f9a48211dd\") "
Nov 03 20:19:50.403775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403692    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-home" pod "3b558ab4-c66b-48b8-a93d-c8f9a48211dd" (UID: "3b558ab4-c66b-48b8-a93d-c8f9a48211dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:50.403975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.403952    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-model-cache" pod "3b558ab4-c66b-48b8-a93d-c8f9a48211dd" (UID: "3b558ab4-c66b-48b8-a93d-c8f9a48211dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:50.404512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.404486    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tmp-dir" pod "3b558ab4-c66b-48b8-a93d-c8f9a48211dd" (UID: "3b558ab4-c66b-48b8-a93d-c8f9a48211dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:50.404955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.404929    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:50.405085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.405070    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:50.405188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.405175    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:50.406687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.406664    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-dshm" pod "3b558ab4-c66b-48b8-a93d-c8f9a48211dd" (UID: "3b558ab4-c66b-48b8-a93d-c8f9a48211dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:50.406961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.406919    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:50.407695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.407669    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tls-certs" pod "3b558ab4-c66b-48b8-a93d-c8f9a48211dd" (UID: "3b558ab4-c66b-48b8-a93d-c8f9a48211dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:50.407767 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.407735    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-kube-api-access-wtlnb" pod "3b558ab4-c66b-48b8-a93d-c8f9a48211dd" (UID: "3b558ab4-c66b-48b8-a93d-c8f9a48211dd"). InnerVolumeSpecName "kube-api-access-wtlnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:50.526223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.509248    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtlnb\" (UniqueName: \"kubernetes.io/projected/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-kube-api-access-wtlnb\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:50.526223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.509292    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:50.526223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.509308    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b558ab4-c66b-48b8-a93d-c8f9a48211dd-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:50.659015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.658922    5282 generic.go:356] "Generic (PLEG): container finished" podID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" containerID="27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375" exitCode=0
Nov 03 20:19:50.659015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.658966    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"
Nov 03 20:19:50.659280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.659008    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" event={"ID":"3b558ab4-c66b-48b8-a93d-c8f9a48211dd","Type":"ContainerDied","Data":"27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375"}
Nov 03 20:19:50.659280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.659051    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6" event={"ID":"3b558ab4-c66b-48b8-a93d-c8f9a48211dd","Type":"ContainerDied","Data":"451b948fb86ed723115727e0a18c0d2738b8ffaa1a893d9162fc4377bbe77d46"}
Nov 03 20:19:50.659280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.659069    5282 scope.go:122] "RemoveContainer" containerID="27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375"
Nov 03 20:19:50.698217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.696441    5282 scope.go:122] "RemoveContainer" containerID="27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375"
Nov 03 20:19:50.702651 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:50.702612    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375\": container with ID starting with 27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375 not found: ID does not exist" containerID="27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375"
Nov 03 20:19:50.702781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.702664    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375"} err="failed to get container status \"27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375\": rpc error: code = NotFound desc = could not find container \"27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375\": container with ID starting with 27d5c3518ae92f5033f2adb729f7267c7227625fcb7c7e97f325bebc80ef3375 not found: ID does not exist"
Nov 03 20:19:50.723881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.723850    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"]
Nov 03 20:19:50.728253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:50.728226    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-663013/test-model-663013-kserve-884b9f4f-zsnh6"]
Nov 03 20:19:51.262271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.255791    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:51.262271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.255999    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:51.333374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.327277    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" podUID="477c28ea-7bed-4489-9bbc-5c0b812b317e" containerName="payload-pre-processing" containerID="cri-o://c9102ac891b92ed4835ff2634bf3b6d7e6019a9c3977be6105070464ddd50d6e" gracePeriod=25
Nov 03 20:19:51.365947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.365917    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:51.457598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.457566    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" path="/var/lib/kubelet/pods/3b558ab4-c66b-48b8-a93d-c8f9a48211dd/volumes"
Nov 03 20:19:51.535338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.535259    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n" podUID="fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" containerName="payload-processing" containerID="cri-o://a2b5fd22d41516800ad9f9cdc01c33a6d523a6ba1b3e13209bd004055b2c5cf3" gracePeriod=25
Nov 03 20:19:51.668698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.667685    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:51.672686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.672658    5282 generic.go:356] "Generic (PLEG): container finished" podID="fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" containerID="a2b5fd22d41516800ad9f9cdc01c33a6d523a6ba1b3e13209bd004055b2c5cf3" exitCode=0
Nov 03 20:19:51.672837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.672728    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n" event={"ID":"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea","Type":"ContainerDied","Data":"a2b5fd22d41516800ad9f9cdc01c33a6d523a6ba1b3e13209bd004055b2c5cf3"}
Nov 03 20:19:51.687087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:51.687049    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:19:52.179672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.179647    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:19:52.205417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.200080    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-plugins-config-volume\") pod \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") "
Nov 03 20:19:52.205417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.200139    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-kube-api-access-qznlf\" (UniqueName: \"kubernetes.io/projected/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-kube-api-access-qznlf\") pod \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\" (UID: \"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea\") "
Nov 03 20:19:52.205417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.200481    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-plugins-config-volume" pod "fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" (UID: "fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:52.205417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.201013    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:52.207181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.207152    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-kube-api-access-qznlf" pod "fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" (UID: "fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea"). InnerVolumeSpecName "kube-api-access-qznlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:52.306967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.306884    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznlf\" (UniqueName: \"kubernetes.io/projected/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea-kube-api-access-qznlf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:52.313453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.313418    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:52.684422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.684332    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"
Nov 03 20:19:52.684422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.684352    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n" event={"ID":"fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea","Type":"ContainerDied","Data":"87c04818ba2cf1b33cc70248d4852e731a9833f76fa93e21f7616b952d03618a"}
Nov 03 20:19:52.684422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.684416    5282 scope.go:122] "RemoveContainer" containerID="a2b5fd22d41516800ad9f9cdc01c33a6d523a6ba1b3e13209bd004055b2c5cf3"
Nov 03 20:19:52.710223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.710179    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"]
Nov 03 20:19:52.715126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:52.715083    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2c6c50-5d96ff4bb6-csn7n"]
Nov 03 20:19:53.352871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:53.352837    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:53.471223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:53.468826    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:53.484227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:53.480572    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" path="/var/lib/kubelet/pods/fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea/volumes"
Nov 03 20:19:53.689405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:53.689319    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"]
Nov 03 20:19:53.689881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:53.689562    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy" containerID="cri-o://0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f" gracePeriod=30
Nov 03 20:19:53.764795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:53.764760    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:54.406904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:54.406869    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:54.992252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:54.992216    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"]
Nov 03 20:19:54.992723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:54.992444    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl" podUID="af36f373-ad1c-474c-845e-95586c8304ef" containerName="maas-api" containerID="cri-o://d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a" gracePeriod=30
Nov 03 20:19:55.066971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.066937    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:55.215247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.212613    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:55.341284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.335801    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:55.526456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.526400    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:55.658432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.658355    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:55.746934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.746891    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-metrics-certs\") pod \"af36f373-ad1c-474c-845e-95586c8304ef\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") "
Nov 03 20:19:55.747108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.746969    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-maas-api-tls\") pod \"af36f373-ad1c-474c-845e-95586c8304ef\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") "
Nov 03 20:19:55.747211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.747151    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af36f373-ad1c-474c-845e-95586c8304ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af36f373-ad1c-474c-845e-95586c8304ef-openshift-service-ca\") pod \"af36f373-ad1c-474c-845e-95586c8304ef\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") "
Nov 03 20:19:55.747595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.747565    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af36f373-ad1c-474c-845e-95586c8304ef-openshift-service-ca" pod "af36f373-ad1c-474c-845e-95586c8304ef" (UID: "af36f373-ad1c-474c-845e-95586c8304ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:55.747701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.747688    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af36f373-ad1c-474c-845e-95586c8304ef-kube-api-access-qbtdf\" (UniqueName: \"kubernetes.io/projected/af36f373-ad1c-474c-845e-95586c8304ef-kube-api-access-qbtdf\") pod \"af36f373-ad1c-474c-845e-95586c8304ef\" (UID: \"af36f373-ad1c-474c-845e-95586c8304ef\") "
Nov 03 20:19:55.748655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.748629    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af36f373-ad1c-474c-845e-95586c8304ef-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:55.749520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.749501    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-maas-api-tls" pod "af36f373-ad1c-474c-845e-95586c8304ef" (UID: "af36f373-ad1c-474c-845e-95586c8304ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:55.749851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.749826    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-metrics-certs" pod "af36f373-ad1c-474c-845e-95586c8304ef" (UID: "af36f373-ad1c-474c-845e-95586c8304ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:55.749974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.749949    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af36f373-ad1c-474c-845e-95586c8304ef-kube-api-access-qbtdf" pod "af36f373-ad1c-474c-845e-95586c8304ef" (UID: "af36f373-ad1c-474c-845e-95586c8304ef"). InnerVolumeSpecName "kube-api-access-qbtdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:55.778924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.778895    5282 generic.go:356] "Generic (PLEG): container finished" podID="af36f373-ad1c-474c-845e-95586c8304ef" containerID="d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a" exitCode=0
Nov 03 20:19:55.779073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.778976    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"
Nov 03 20:19:55.779073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.778992    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl" event={"ID":"af36f373-ad1c-474c-845e-95586c8304ef","Type":"ContainerDied","Data":"d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a"}
Nov 03 20:19:55.779073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.779056    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl" event={"ID":"af36f373-ad1c-474c-845e-95586c8304ef","Type":"ContainerDied","Data":"23072a98702d34834bd8f85160aa24fa0ee847e1cb325e0fa0c874d8e2e23138"}
Nov 03 20:19:55.779271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.779086    5282 scope.go:122] "RemoveContainer" containerID="d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a"
Nov 03 20:19:55.815300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.815272    5282 scope.go:122] "RemoveContainer" containerID="d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a"
Nov 03 20:19:55.815665 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:55.815632    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a\": container with ID starting with d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a not found: ID does not exist" containerID="d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a"
Nov 03 20:19:55.815792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.815707    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a"} err="failed to get container status \"d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a\": rpc error: code = NotFound desc = could not find container \"d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a\": container with ID starting with d1588fd808b7dccb3487c3162f5249aa270cef5f62b8122552f1ad38ac41bb9a not found: ID does not exist"
Nov 03 20:19:55.819190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.819154    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"]
Nov 03 20:19:55.823482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.823461    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl"]
Nov 03 20:19:55.850833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.850800    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:55.850833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.850828    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af36f373-ad1c-474c-845e-95586c8304ef-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:55.850993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.850840    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbtdf\" (UniqueName: \"kubernetes.io/projected/af36f373-ad1c-474c-845e-95586c8304ef-kube-api-access-qbtdf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:55.886711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:55.886676    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:56.184235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:56.183872    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:56.359218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:56.353661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:56.803549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:56.803515    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:56.927219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:56.923332    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:57.218739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.218711    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:57.457711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.457621    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af36f373-ad1c-474c-845e-95586c8304ef" path="/var/lib/kubelet/pods/af36f373-ad1c-474c-845e-95586c8304ef/volumes"
Nov 03 20:19:57.472877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.472850    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:57.802130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.802099    5282 generic.go:356] "Generic (PLEG): container finished" podID="de77257b-93d9-472b-a007-a6d199acf1e2" containerID="6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634" exitCode=0
Nov 03 20:19:57.802356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.802173    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" event={"ID":"de77257b-93d9-472b-a007-a6d199acf1e2","Type":"ContainerDied","Data":"6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634"}
Nov 03 20:19:57.802735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.802717    5282 scope.go:122] "RemoveContainer" containerID="6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634"
Nov 03 20:19:57.804765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.804735    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:57.804847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.804805    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:57.804847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:57.804841    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:19:58.394254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:58.394208    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:58.506232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:58.504306    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2c6c50-64fdd59c85-4cvwl" podUID="af36f373-ad1c-474c-845e-95586c8304ef" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.173:8443/health\": context deadline exceeded"
Nov 03 20:19:58.814132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:58.814099    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" event={"ID":"de77257b-93d9-472b-a007-a6d199acf1e2","Type":"ContainerStarted","Data":"8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7"}
Nov 03 20:19:58.924219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:58.920748    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"]
Nov 03 20:19:58.927377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:58.924652    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"]
Nov 03 20:19:59.019675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.019624    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:59.104241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.100875    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.151743    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176364    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-kube-api-access-nwhmj\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-kube-api-access-nwhmj\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176466    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-data\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176516    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-podinfo\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176552    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-token\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-token\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176594    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istiod-ca-cert\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176682    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-certs\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176727    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-credential-socket\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176761    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-socket\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.187300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.176899    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-envoy\") pod \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\" (UID: \"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe\") "
Nov 03 20:19:59.221248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.218603    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-data" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.221761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.221508    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-certs" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.226969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.221994    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istiod-ca-cert" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:59.233258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.227514    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-socket" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.245297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.233464    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-credential-socket" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.296298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.294183    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.296298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.294246    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.296298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.294265    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.296298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.294279    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.296298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.294293    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.352293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.336383    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-podinfo" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:59.451605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.434492    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.451605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.449145    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-kube-api-access-nwhmj" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "kube-api-access-nwhmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:59.469225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.462364    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-envoy" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:59.495537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.475361    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-token" pod "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" (UID: "1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:59.551371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.551041    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.551371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.551073    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhmj\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-kube-api-access-nwhmj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.551371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.551084    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:59.669460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.669428    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:19:59.670142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.670110    5282 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:19:59.670306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.670185    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 20:19:59.823990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.823889    5282 generic.go:356] "Generic (PLEG): container finished" podID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerID="0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f" exitCode=0
Nov 03 20:19:59.823990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.823942    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"
Nov 03 20:19:59.823990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.823978    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" event={"ID":"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe","Type":"ContainerDied","Data":"0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f"}
Nov 03 20:19:59.824317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.824030    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg" event={"ID":"1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe","Type":"ContainerDied","Data":"1ae5a8d85fa1bddf94a22cf1c80ea1e38d50e0767dbc3b0fd851736e515ff731"}
Nov 03 20:19:59.824317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.824051    5282 scope.go:122] "RemoveContainer" containerID="0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f"
Nov 03 20:19:59.835357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.835338    5282 scope.go:122] "RemoveContainer" containerID="0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f"
Nov 03 20:19:59.835639 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:19:59.835622    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f\": container with ID starting with 0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f not found: ID does not exist" containerID="0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f"
Nov 03 20:19:59.835679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.835647    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f"} err="failed to get container status \"0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f\": rpc error: code = NotFound desc = could not find container \"0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f\": container with ID starting with 0e1daf4e65947abb60dd09009c06cda7d08178ef862c0ee12f89b7869c54bc4f not found: ID does not exist"
Nov 03 20:19:59.845052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.845026    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"]
Nov 03 20:19:59.849374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.849348    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-9txlg"]
Nov 03 20:19:59.968671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:19:59.968640    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:00.553928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:00.553889    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:00.674222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:00.672336    5282 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:20:00.674222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:00.672434    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 20:20:00.700282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:00.700233    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:00.920338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:00.917033    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:01.144220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:01.143457    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:01.320298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:01.320251    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" podUID="477c28ea-7bed-4489-9bbc-5c0b812b317e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:01.429217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:01.421415    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:01.589277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:01.573313    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" path="/var/lib/kubelet/pods/1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe/volumes"
Nov 03 20:20:01.651133 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:01.651098    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:20:01.670578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:01.670545    5282 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:20:01.670728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:01.670603    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 20:20:02.111583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.111545    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"]
Nov 03 20:20:02.119368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.119340    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:02.162450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.162421    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:20:02.319167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.319123    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:02.454274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.446256    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:02.748255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.746181    5282 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:20:02.748255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.746268    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 20:20:02.758243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.755402    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:02.764741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.764711    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"]
Nov 03 20:20:02.765116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.764951    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" containerID="cri-o://14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1" gracePeriod=30
Nov 03 20:20:02.770735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.770711    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"]
Nov 03 20:20:02.771130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:02.771102    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" containerID="cri-o://347d98c8e88fe80a339fed93aa1e89b09546027c3a9c6140bfc7c35d827d6a5a" gracePeriod=30
Nov 03 20:20:03.042974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.042354    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:03.042974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.042469    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:03.127513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.106644    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"]
Nov 03 20:20:03.127513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.106922    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" containerID="cri-o://cda6307b4bcf70d1581e8da93a7cd4f33cde5449f55c47814dd61a7fe8f69f82" gracePeriod=30
Nov 03 20:20:03.306498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.306394    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:03.674576 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.671616    5282 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:20:03.674576 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.671682    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 20:20:03.944939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.944837    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" podUID="7b4103a6-0edc-448e-b349-7679568bf8d2" containerName="payload-processing" containerID="cri-o://ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546" gracePeriod=25
Nov 03 20:20:03.948618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.948588    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" podUID="358e261d-eb63-493c-bc15-8c202ee4af7f" containerName="payload-pre-processing" containerID="cri-o://c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6" gracePeriod=25
Nov 03 20:20:03.949522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:03.949494    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:04.095221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.086858    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:04.258221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.240329    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:04.483930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.475118    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:04.580303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.565733    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"]
Nov 03 20:20:04.580303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.565999    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy" containerID="cri-o://504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878" gracePeriod=30
Nov 03 20:20:04.636509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.636318    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:04.690240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.677234    5282 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:20:04.690240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.677308    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 20:20:04.753041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:04.751154    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" podUID="358e261d-eb63-493c-bc15-8c202ee4af7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:05.399537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:05.399498    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:05.670324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:05.670243    5282 patch_prober.go:29] interesting pod/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:20:05.670324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:05.670313    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 20:20:05.722244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:05.722169    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" podUID="7b4103a6-0edc-448e-b349-7679568bf8d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:06.214531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:06.214475    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:06.458989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:06.458955    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:06.675053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:06.675022    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:20:06.676092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:06.676070    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:20:07.073003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:07.072966    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:07.114289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:07.114216    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" podUID="030e2c8d-47b1-4787-9efb-393780359cc1" containerName="payload-pre-processing" containerID="cri-o://34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c" gracePeriod=25
Nov 03 20:20:07.385024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:07.384935    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:07.385024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:07.384995    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:07.629573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:07.629522    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:07.630067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:07.629622    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:07.706482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:07.706382    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:08.085372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.085341    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:20:08.178175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178138    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-certs\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178185    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-podinfo\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178361    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-socket\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178444    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-credential-socket\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178570    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-data\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178641    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-kube-api-access-wmchq\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-kube-api-access-wmchq\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178668    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f48c89-4498-48ff-8c32-4a2c796151f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f48c89-4498-48ff-8c32-4a2c796151f1-istiod-ca-cert\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178709    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-token\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-token\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.178836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.178734    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-envoy\") pod \"33f48c89-4498-48ff-8c32-4a2c796151f1\" (UID: \"33f48c89-4498-48ff-8c32-4a2c796151f1\") "
Nov 03 20:20:08.179240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.179166    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-credential-socket" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.180003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.179533    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-certs" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.180003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.179703    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-socket" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.180003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.179765    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.180003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.179794    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.180674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.180638    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f48c89-4498-48ff-8c32-4a2c796151f1-istiod-ca-cert" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:08.180979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.180954    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-data" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.183418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.183392    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-podinfo" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:08.183418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.183405    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-envoy" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.183590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.183426    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-token" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.184922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.184896    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-kube-api-access-wmchq" pod "33f48c89-4498-48ff-8c32-4a2c796151f1" (UID: "33f48c89-4498-48ff-8c32-4a2c796151f1"). InnerVolumeSpecName "kube-api-access-wmchq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.282819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.282787    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.282972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.282823    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmchq\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-kube-api-access-wmchq\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.282972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.282840    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f48c89-4498-48ff-8c32-4a2c796151f1-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.282972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.282853    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.282972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.282867    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.282972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.282881    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f48c89-4498-48ff-8c32-4a2c796151f1-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.282972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.282895    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f48c89-4498-48ff-8c32-4a2c796151f1-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.417105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.416797    5282 generic.go:356] "Generic (PLEG): container finished" podID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerID="cda6307b4bcf70d1581e8da93a7cd4f33cde5449f55c47814dd61a7fe8f69f82" exitCode=0
Nov 03 20:20:08.417105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.416995    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" event={"ID":"55c2b3db-5451-4b5a-8313-9fc3fb58db6b","Type":"ContainerDied","Data":"cda6307b4bcf70d1581e8da93a7cd4f33cde5449f55c47814dd61a7fe8f69f82"}
Nov 03 20:20:08.435023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.434987    5282 generic.go:356] "Generic (PLEG): container finished" podID="67151051-1ed2-4a57-a34c-4465e6538648" containerID="347d98c8e88fe80a339fed93aa1e89b09546027c3a9c6140bfc7c35d827d6a5a" exitCode=0
Nov 03 20:20:08.435216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.435125    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" event={"ID":"67151051-1ed2-4a57-a34c-4465e6538648","Type":"ContainerDied","Data":"347d98c8e88fe80a339fed93aa1e89b09546027c3a9c6140bfc7c35d827d6a5a"}
Nov 03 20:20:08.436651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.436622    5282 generic.go:356] "Generic (PLEG): container finished" podID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerID="14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1" exitCode=0
Nov 03 20:20:08.436830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.436814    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" event={"ID":"33f48c89-4498-48ff-8c32-4a2c796151f1","Type":"ContainerDied","Data":"14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1"}
Nov 03 20:20:08.436922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.436860    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6" event={"ID":"33f48c89-4498-48ff-8c32-4a2c796151f1","Type":"ContainerDied","Data":"a348531e6a7be7896843c4f4d2f4d3a6ecfa806645863d3bd3d79fe567c7bc80"}
Nov 03 20:20:08.436922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.436897    5282 scope.go:122] "RemoveContainer" containerID="14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1"
Nov 03 20:20:08.437083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.437067    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"
Nov 03 20:20:08.470504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.470472    5282 scope.go:122] "RemoveContainer" containerID="14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1"
Nov 03 20:20:08.471182 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:08.471146    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1\": container with ID starting with 14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1 not found: ID does not exist" containerID="14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1"
Nov 03 20:20:08.471322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.471229    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1"} err="failed to get container status \"14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1\": rpc error: code = NotFound desc = could not find container \"14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1\": container with ID starting with 14a6ecdb994b6245235afc557b35788e2a45f8b9fabc65603e6fd91061fc17c1 not found: ID does not exist"
Nov 03 20:20:08.478259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.477430    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"]
Nov 03 20:20:08.483948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.483919    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-fs6r6"]
Nov 03 20:20:08.490852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.490788    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:08.629363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.628740    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:08.722231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.717843    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:20:08.755566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.755527    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"]
Nov 03 20:20:08.755809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.755783    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy" containerID="cri-o://02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088" gracePeriod=30
Nov 03 20:20:08.800757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.800728    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:20:08.810289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810262    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-credential-socket\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.810456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810313    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-token\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.810456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810348    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-podinfo\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.810456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810398    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istiod-ca-cert\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.810694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810473    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-certs\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.810694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810507    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-envoy\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.810906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810881    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istiod-ca-cert" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:08.810978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810884    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-kube-api-access-d5cb8\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-kube-api-access-d5cb8\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.810978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810953    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-data\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.811098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.810988    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-socket\") pod \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\" (UID: \"55c2b3db-5451-4b5a-8313-9fc3fb58db6b\") "
Nov 03 20:20:08.811969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.811945    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:08.812276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.812247    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-socket" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.812486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.812461    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-certs" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.812756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.812738    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-data" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.812957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.812940    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-credential-socket" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.814607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.814580    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-kube-api-access-d5cb8" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "kube-api-access-d5cb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.816516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.816493    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-podinfo" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:08.816620 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.816512    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-envoy" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:08.817759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.817735    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-token" pod "55c2b3db-5451-4b5a-8313-9fc3fb58db6b" (UID: "55c2b3db-5451-4b5a-8313-9fc3fb58db6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:08.824068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.824044    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.998810    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-credential-socket\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.998942    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67151051-1ed2-4a57-a34c-4465e6538648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67151051-1ed2-4a57-a34c-4465e6538648-istiod-ca-cert\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.999035    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-envoy\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.999085    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-socket\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.999161    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-certs\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.999223    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-data\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.999270    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-istio-token\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-istio-token\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.999318    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-kube-api-access-4qm6v\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-kube-api-access-4qm6v\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:08.999408    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67151051-1ed2-4a57-a34c-4465e6538648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67151051-1ed2-4a57-a34c-4465e6538648-istio-podinfo\") pod \"67151051-1ed2-4a57-a34c-4465e6538648\" (UID: \"67151051-1ed2-4a57-a34c-4465e6538648\") "
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000251    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5cb8\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-kube-api-access-d5cb8\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000283    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000298    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000313    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000328    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000342    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000358    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.002303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.000372    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c2b3db-5451-4b5a-8313-9fc3fb58db6b-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.024225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.003504    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-socket" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:09.024225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.008285    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-credential-socket" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:09.024225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.012431    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67151051-1ed2-4a57-a34c-4465e6538648-istiod-ca-cert" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:09.051292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.044355    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-certs" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:09.051292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.045180    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67151051-1ed2-4a57-a34c-4465e6538648-istio-podinfo" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:09.051292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.046287    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-data" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:09.113223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.106598    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67151051-1ed2-4a57-a34c-4465e6538648-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.113223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.106634    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.113223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.106652    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67151051-1ed2-4a57-a34c-4465e6538648-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.113223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.106666    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.113223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.106679    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.113223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.106693    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.165850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.163868    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-kube-api-access-4qm6v" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "kube-api-access-4qm6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:09.180728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.180682    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-envoy" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:09.182359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.182322    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-istio-token" pod "67151051-1ed2-4a57-a34c-4465e6538648" (UID: "67151051-1ed2-4a57-a34c-4465e6538648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:09.208246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.208217    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67151051-1ed2-4a57-a34c-4465e6538648-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.208246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.208242    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.208438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.208252    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qm6v\" (UniqueName: \"kubernetes.io/projected/67151051-1ed2-4a57-a34c-4465e6538648-kube-api-access-4qm6v\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:09.382181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.382149    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"]
Nov 03 20:20:09.401997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.401966    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"]
Nov 03 20:20:09.417032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.417004    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"]
Nov 03 20:20:09.454135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.454013    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"
Nov 03 20:20:09.456562 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.456525    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"
Nov 03 20:20:09.462741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.462711    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" path="/var/lib/kubelet/pods/33f48c89-4498-48ff-8c32-4a2c796151f1/volumes"
Nov 03 20:20:09.463695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.463674    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm" event={"ID":"55c2b3db-5451-4b5a-8313-9fc3fb58db6b","Type":"ContainerDied","Data":"38a19c0cd8e00feb804c1251219fb869e2ac3bbcfe6005eac88aec63b98f36d3"}
Nov 03 20:20:09.463753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.463729    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg" event={"ID":"67151051-1ed2-4a57-a34c-4465e6538648","Type":"ContainerDied","Data":"9aa0524e1c6d47ff231697444c4e99ff52993d5afe402f62ae09ca8212573e18"}
Nov 03 20:20:09.463795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.463783    5282 scope.go:122] "RemoveContainer" containerID="cda6307b4bcf70d1581e8da93a7cd4f33cde5449f55c47814dd61a7fe8f69f82"
Nov 03 20:20:09.478081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.478065    5282 scope.go:122] "RemoveContainer" containerID="347d98c8e88fe80a339fed93aa1e89b09546027c3a9c6140bfc7c35d827d6a5a"
Nov 03 20:20:09.492710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.492677    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"]
Nov 03 20:20:09.496950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.496929    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-mkdsg"]
Nov 03 20:20:09.507323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.507270    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"]
Nov 03 20:20:09.514766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.514740    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2c6c50-openshift-default-6f8d9f9f77-dv4dm"]
Nov 03 20:20:09.534281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.534256    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:09.676221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.664896    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:09.933091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:09.933062    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:20:10.024215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024175    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-podinfo\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024299    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-envoy\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024343    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-certs\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024466    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-socket\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024584    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-credential-socket\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024613    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-data\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024665    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60336029-d74c-4227-a77f-7f2bfc2a17f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60336029-d74c-4227-a77f-7f2bfc2a17f1-istiod-ca-cert\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024707    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-certs" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:10.024840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024722    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-socket" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:10.024840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024780    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-kube-api-access-wcxn7\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-kube-api-access-wcxn7\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.024951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024853    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-credential-socket" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:10.024951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024885    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-token\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-token\") pod \"60336029-d74c-4227-a77f-7f2bfc2a17f1\" (UID: \"60336029-d74c-4227-a77f-7f2bfc2a17f1\") "
Nov 03 20:20:10.025065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.024943    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-data" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:10.025122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.025075    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60336029-d74c-4227-a77f-7f2bfc2a17f1-istiod-ca-cert" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:10.025612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.025590    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.025612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.025611    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.025801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.025621    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.025801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.025631    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.025801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.025639    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60336029-d74c-4227-a77f-7f2bfc2a17f1-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.026584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.026560    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-podinfo" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:10.027000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.026978    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-envoy" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:10.027085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.027021    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-kube-api-access-wcxn7" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "kube-api-access-wcxn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:10.027301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.027285    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-token" pod "60336029-d74c-4227-a77f-7f2bfc2a17f1" (UID: "60336029-d74c-4227-a77f-7f2bfc2a17f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:10.127790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.127708    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxn7\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-kube-api-access-wcxn7\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.127790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.127739    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.127790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.127750    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.127790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.127761    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60336029-d74c-4227-a77f-7f2bfc2a17f1-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:10.468430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.468335    5282 generic.go:356] "Generic (PLEG): container finished" podID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerID="504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878" exitCode=0
Nov 03 20:20:10.468430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.468395    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" event={"ID":"60336029-d74c-4227-a77f-7f2bfc2a17f1","Type":"ContainerDied","Data":"504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878"}
Nov 03 20:20:10.468430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.468416    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"
Nov 03 20:20:10.468689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.468447    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q" event={"ID":"60336029-d74c-4227-a77f-7f2bfc2a17f1","Type":"ContainerDied","Data":"3e78e078c58660aa65f14d083a1902f0b0c52f42043eeecdbafe6521e21bc299"}
Nov 03 20:20:10.468689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.468478    5282 scope.go:122] "RemoveContainer" containerID="504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878"
Nov 03 20:20:10.480687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.480667    5282 scope.go:122] "RemoveContainer" containerID="2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172"
Nov 03 20:20:10.492066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.492040    5282 scope.go:122] "RemoveContainer" containerID="504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878"
Nov 03 20:20:10.492403 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:10.492383    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878\": container with ID starting with 504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878 not found: ID does not exist" containerID="504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878"
Nov 03 20:20:10.492459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.492413    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878"} err="failed to get container status \"504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878\": rpc error: code = NotFound desc = could not find container \"504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878\": container with ID starting with 504487d76bd09ecd1f22f1eca4db1c6085f212719ccec724fe7fb5b73f394878 not found: ID does not exist"
Nov 03 20:20:10.492459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.492429    5282 scope.go:122] "RemoveContainer" containerID="2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172"
Nov 03 20:20:10.492688 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:10.492669    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172\": container with ID starting with 2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172 not found: ID does not exist" containerID="2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172"
Nov 03 20:20:10.492741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.492696    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172"} err="failed to get container status \"2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172\": rpc error: code = NotFound desc = could not find container \"2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172\": container with ID starting with 2b5ccef3aa24f433533b749f542a88fe86910b21451325ad4c4575dd6ff9d172 not found: ID does not exist"
Nov 03 20:20:10.512563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.512532    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"]
Nov 03 20:20:10.541419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.541384    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f817b0-openshift-default-64d96d4677-h6j9q"]
Nov 03 20:20:10.598450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:10.598401    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:11.258444 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.258407    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:11.284339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.284306    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" podUID="477c28ea-7bed-4489-9bbc-5c0b812b317e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:11.449155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.449126    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:11.458067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.458038    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" path="/var/lib/kubelet/pods/55c2b3db-5451-4b5a-8313-9fc3fb58db6b/volumes"
Nov 03 20:20:11.458583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.458568    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" path="/var/lib/kubelet/pods/60336029-d74c-4227-a77f-7f2bfc2a17f1/volumes"
Nov 03 20:20:11.459029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.459016    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67151051-1ed2-4a57-a34c-4465e6538648" path="/var/lib/kubelet/pods/67151051-1ed2-4a57-a34c-4465e6538648/volumes"
Nov 03 20:20:11.479551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.479522    5282 generic.go:356] "Generic (PLEG): container finished" podID="477c28ea-7bed-4489-9bbc-5c0b812b317e" containerID="c9102ac891b92ed4835ff2634bf3b6d7e6019a9c3977be6105070464ddd50d6e" exitCode=0
Nov 03 20:20:11.479740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.479604    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" event={"ID":"477c28ea-7bed-4489-9bbc-5c0b812b317e","Type":"ContainerDied","Data":"c9102ac891b92ed4835ff2634bf3b6d7e6019a9c3977be6105070464ddd50d6e"}
Nov 03 20:20:11.552313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.552280    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:11.866317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.862927    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:11.898019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.897985    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:20:11.985089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:11.985055    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:12.056391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.056355    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477c28ea-7bed-4489-9bbc-5c0b812b317e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477c28ea-7bed-4489-9bbc-5c0b812b317e-plugins-config-volume\") pod \"477c28ea-7bed-4489-9bbc-5c0b812b317e\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") "
Nov 03 20:20:12.056610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.056589    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477c28ea-7bed-4489-9bbc-5c0b812b317e-kube-api-access-bbs4g\" (UniqueName: \"kubernetes.io/projected/477c28ea-7bed-4489-9bbc-5c0b812b317e-kube-api-access-bbs4g\") pod \"477c28ea-7bed-4489-9bbc-5c0b812b317e\" (UID: \"477c28ea-7bed-4489-9bbc-5c0b812b317e\") "
Nov 03 20:20:12.058034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.058003    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477c28ea-7bed-4489-9bbc-5c0b812b317e-plugins-config-volume" pod "477c28ea-7bed-4489-9bbc-5c0b812b317e" (UID: "477c28ea-7bed-4489-9bbc-5c0b812b317e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:12.061682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.061658    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477c28ea-7bed-4489-9bbc-5c0b812b317e-kube-api-access-bbs4g" pod "477c28ea-7bed-4489-9bbc-5c0b812b317e" (UID: "477c28ea-7bed-4489-9bbc-5c0b812b317e"). InnerVolumeSpecName "kube-api-access-bbs4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:12.149143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.149063    5282 patch_prober.go:29] interesting pod/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:12.149143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.149122    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:12.158906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.158873    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbs4g\" (UniqueName: \"kubernetes.io/projected/477c28ea-7bed-4489-9bbc-5c0b812b317e-kube-api-access-bbs4g\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:12.158906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.158904    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477c28ea-7bed-4489-9bbc-5c0b812b317e-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:12.488668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.488635    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"
Nov 03 20:20:12.489118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.488679    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk" event={"ID":"477c28ea-7bed-4489-9bbc-5c0b812b317e","Type":"ContainerDied","Data":"f27e9f6c3a573bdf1d91263bf0af8721fcf2b3c63ea190f3858bc9925ac088c8"}
Nov 03 20:20:12.489118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.488749    5282 scope.go:122] "RemoveContainer" containerID="c9102ac891b92ed4835ff2634bf3b6d7e6019a9c3977be6105070464ddd50d6e"
Nov 03 20:20:12.519827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.519798    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"]
Nov 03 20:20:12.527530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.527502    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f817b0-8784455dc-kv8tk"]
Nov 03 20:20:12.603439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:12.603401    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:13.466088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:13.466052    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477c28ea-7bed-4489-9bbc-5c0b812b317e" path="/var/lib/kubelet/pods/477c28ea-7bed-4489-9bbc-5c0b812b317e/volumes"
Nov 03 20:20:13.561760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:13.561723    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:14.054656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.054628    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:20:14.130703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.130665    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-envoy\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.130888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.130739    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-credential-socket\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.130888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.130776    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-kube-api-access-479p4\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-kube-api-access-479p4\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.130888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.130817    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be91e70c-072f-4fab-be70-d26878da83e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be91e70c-072f-4fab-be70-d26878da83e0-istio-podinfo\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.131079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.130888    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-socket\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.131079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.130915    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be91e70c-072f-4fab-be70-d26878da83e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be91e70c-072f-4fab-be70-d26878da83e0-istiod-ca-cert\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.131079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.130971    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-istio-token\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-istio-token\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.131079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.131036    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-credential-socket" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:14.131311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.131163    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-certs\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.131311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.131274    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-data\") pod \"be91e70c-072f-4fab-be70-d26878da83e0\" (UID: \"be91e70c-072f-4fab-be70-d26878da83e0\") "
Nov 03 20:20:14.131414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.131335    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-socket" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:14.131505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.131481    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be91e70c-072f-4fab-be70-d26878da83e0-istiod-ca-cert" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:14.131585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.131564    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-data" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:14.131744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.131716    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-certs" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:14.132238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.132177    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.132238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.132216    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.132238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.132230    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be91e70c-072f-4fab-be70-d26878da83e0-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.132238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.132239    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.132539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.132247    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.133148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.133126    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-envoy" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:14.133402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.133383    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-kube-api-access-479p4" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "kube-api-access-479p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:14.134533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.133767    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-istio-token" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:14.134533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.133789    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be91e70c-072f-4fab-be70-d26878da83e0-istio-podinfo" pod "be91e70c-072f-4fab-be70-d26878da83e0" (UID: "be91e70c-072f-4fab-be70-d26878da83e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:14.234124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.234077    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be91e70c-072f-4fab-be70-d26878da83e0-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.234124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.234119    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479p4\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-kube-api-access-479p4\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.234430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.234141    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be91e70c-072f-4fab-be70-d26878da83e0-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.234430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.234154    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be91e70c-072f-4fab-be70-d26878da83e0-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.272000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.271917    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:14.507316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.507283    5282 generic.go:356] "Generic (PLEG): container finished" podID="be91e70c-072f-4fab-be70-d26878da83e0" containerID="02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088" exitCode=0
Nov 03 20:20:14.507502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.507340    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"
Nov 03 20:20:14.507502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.507354    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" event={"ID":"be91e70c-072f-4fab-be70-d26878da83e0","Type":"ContainerDied","Data":"02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088"}
Nov 03 20:20:14.507502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.507394    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md" event={"ID":"be91e70c-072f-4fab-be70-d26878da83e0","Type":"ContainerDied","Data":"865aed29313fc2b351fa8961f451cca5121dd0a158a5a8cd05972fea92d5c17a"}
Nov 03 20:20:14.507502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.507416    5282 scope.go:122] "RemoveContainer" containerID="02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088"
Nov 03 20:20:14.520621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.520595    5282 scope.go:122] "RemoveContainer" containerID="11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4"
Nov 03 20:20:14.531101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.531082    5282 scope.go:122] "RemoveContainer" containerID="02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088"
Nov 03 20:20:14.531412 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:14.531383    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088\": container with ID starting with 02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088 not found: ID does not exist" containerID="02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088"
Nov 03 20:20:14.531513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.531418    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088"} err="failed to get container status \"02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088\": rpc error: code = NotFound desc = could not find container \"02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088\": container with ID starting with 02c876a4dc3c9db7df85f5b0be988d6b5b239444a6927de6b7849aabf3194088 not found: ID does not exist"
Nov 03 20:20:14.531513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.531437    5282 scope.go:122] "RemoveContainer" containerID="11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4"
Nov 03 20:20:14.531763 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:14.531736    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4\": container with ID starting with 11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4 not found: ID does not exist" containerID="11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4"
Nov 03 20:20:14.531810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.531772    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4"} err="failed to get container status \"11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4\": rpc error: code = NotFound desc = could not find container \"11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4\": container with ID starting with 11b2106803643283f4168f0a6fb89b3763d2d6a0027002645981da4dc23a7af4 not found: ID does not exist"
Nov 03 20:20:14.545021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.544982    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"]
Nov 03 20:20:14.551684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.551658    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2092ba-openshift-default-5dd4c7f4dc-w46md"]
Nov 03 20:20:14.657616 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.657581    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" podUID="358e261d-eb63-493c-bc15-8c202ee4af7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:14.818172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.818090    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"]
Nov 03 20:20:14.956682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:14.956648    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:15.107286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:15.106868    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:15.243221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:15.241935    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:15.482979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:15.482942    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be91e70c-072f-4fab-be70-d26878da83e0" path="/var/lib/kubelet/pods/be91e70c-072f-4fab-be70-d26878da83e0/volumes"
Nov 03 20:20:15.721536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:15.721497    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" podUID="7b4103a6-0edc-448e-b349-7679568bf8d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:16.192220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:16.187656    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:16.640513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:16.640471    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" podUID="030e2c8d-47b1-4787-9efb-393780359cc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:16.826547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:16.826510    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:17.033107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:17.033070    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:17.916389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:17.916342    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:18.649698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:18.645839    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:18.877094 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:18.877062    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:19.002223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:19.001953    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:19.530183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:19.530146    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"]
Nov 03 20:20:19.541343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:19.541312    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"]
Nov 03 20:20:19.635752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:19.635706    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:19.749782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:19.749751    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:19.819046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:19.818962    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb" podUID="ae51af71-0640-4d6d-bec2-f819ef9ebad6" containerName="payload-processing" containerID="cri-o://cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5" gracePeriod=25
Nov 03 20:20:19.903222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:19.901462    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:20.284110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.284081    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:20:20.405614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.405582    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae51af71-0640-4d6d-bec2-f819ef9ebad6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae51af71-0640-4d6d-bec2-f819ef9ebad6-plugins-config-volume\") pod \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") "
Nov 03 20:20:20.405782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.405750    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae51af71-0640-4d6d-bec2-f819ef9ebad6-kube-api-access-6d8kb\" (UniqueName: \"kubernetes.io/projected/ae51af71-0640-4d6d-bec2-f819ef9ebad6-kube-api-access-6d8kb\") pod \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\" (UID: \"ae51af71-0640-4d6d-bec2-f819ef9ebad6\") "
Nov 03 20:20:20.406049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.406020    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae51af71-0640-4d6d-bec2-f819ef9ebad6-plugins-config-volume" pod "ae51af71-0640-4d6d-bec2-f819ef9ebad6" (UID: "ae51af71-0640-4d6d-bec2-f819ef9ebad6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:20.406456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.406431    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae51af71-0640-4d6d-bec2-f819ef9ebad6-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:20.407967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.407947    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae51af71-0640-4d6d-bec2-f819ef9ebad6-kube-api-access-6d8kb" pod "ae51af71-0640-4d6d-bec2-f819ef9ebad6" (UID: "ae51af71-0640-4d6d-bec2-f819ef9ebad6"). InnerVolumeSpecName "kube-api-access-6d8kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:20.507842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.507796    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8kb\" (UniqueName: \"kubernetes.io/projected/ae51af71-0640-4d6d-bec2-f819ef9ebad6-kube-api-access-6d8kb\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:20.564254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.564188    5282 generic.go:356] "Generic (PLEG): container finished" podID="ae51af71-0640-4d6d-bec2-f819ef9ebad6" containerID="cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5" exitCode=0
Nov 03 20:20:20.564428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.564275    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"
Nov 03 20:20:20.564428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.564271    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb" event={"ID":"ae51af71-0640-4d6d-bec2-f819ef9ebad6","Type":"ContainerDied","Data":"cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5"}
Nov 03 20:20:20.564428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.564402    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb" event={"ID":"ae51af71-0640-4d6d-bec2-f819ef9ebad6","Type":"ContainerDied","Data":"5608cdc109b60d69cb70e93a532553565d157f25a303cff931c8e63211eb5f08"}
Nov 03 20:20:20.564428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.564425    5282 scope.go:122] "RemoveContainer" containerID="cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5"
Nov 03 20:20:20.578722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.578703    5282 scope.go:122] "RemoveContainer" containerID="cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5"
Nov 03 20:20:20.579143 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:20.579118    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5\": container with ID starting with cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5 not found: ID does not exist" containerID="cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5"
Nov 03 20:20:20.579242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.579156    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5"} err="failed to get container status \"cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5\": rpc error: code = NotFound desc = could not find container \"cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5\": container with ID starting with cf557f6c38264f6ac87042995b151f207724f9d7fc988834b0fea17c91d145b5 not found: ID does not exist"
Nov 03 20:20:20.593237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.592350    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"]
Nov 03 20:20:20.597997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.597974    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-5348ca-87b665cfd-wjfgb"]
Nov 03 20:20:20.709607 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:20.709482    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:21.320290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:21.320230    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:21.457030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:21.456998    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae51af71-0640-4d6d-bec2-f819ef9ebad6" path="/var/lib/kubelet/pods/ae51af71-0640-4d6d-bec2-f819ef9ebad6/volumes"
Nov 03 20:20:21.576345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:21.576240    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:21.695908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:21.695875    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:22.113082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:22.113045    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:22.881329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:22.881292    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:23.331187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.331150    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"]
Nov 03 20:20:23.331735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.331701    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6" podUID="deaba6ad-8109-48f9-81ca-62e226deff03" containerName="maas-api" containerID="cri-o://d1a93542c7e4a9d6031960099b6b96aeae36beacb0c4049cace18211b78d4e9b" gracePeriod=30
Nov 03 20:20:23.359601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.359556    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6" podUID="deaba6ad-8109-48f9-81ca-62e226deff03" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.150:8443/health\": dial tcp 10.133.0.150:8443: connect: connection refused"
Nov 03 20:20:23.390746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.390704    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:23.530781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.529710    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:23.597405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.597319    5282 generic.go:356] "Generic (PLEG): container finished" podID="deaba6ad-8109-48f9-81ca-62e226deff03" containerID="d1a93542c7e4a9d6031960099b6b96aeae36beacb0c4049cace18211b78d4e9b" exitCode=0
Nov 03 20:20:23.597567 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.597396    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6" event={"ID":"deaba6ad-8109-48f9-81ca-62e226deff03","Type":"ContainerDied","Data":"d1a93542c7e4a9d6031960099b6b96aeae36beacb0c4049cace18211b78d4e9b"}
Nov 03 20:20:23.632507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.632483    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:20:23.641355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.641321    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:23.757072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.757039    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:23.804218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.799369    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-metrics-certs\") pod \"deaba6ad-8109-48f9-81ca-62e226deff03\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") "
Nov 03 20:20:23.804218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.799572    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-maas-api-tls\") pod \"deaba6ad-8109-48f9-81ca-62e226deff03\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") "
Nov 03 20:20:23.804218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.799719    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deaba6ad-8109-48f9-81ca-62e226deff03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaba6ad-8109-48f9-81ca-62e226deff03-openshift-service-ca\") pod \"deaba6ad-8109-48f9-81ca-62e226deff03\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") "
Nov 03 20:20:23.804218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.799807    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deaba6ad-8109-48f9-81ca-62e226deff03-kube-api-access-788mm\" (UniqueName: \"kubernetes.io/projected/deaba6ad-8109-48f9-81ca-62e226deff03-kube-api-access-788mm\") pod \"deaba6ad-8109-48f9-81ca-62e226deff03\" (UID: \"deaba6ad-8109-48f9-81ca-62e226deff03\") "
Nov 03 20:20:23.813219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.809894    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaba6ad-8109-48f9-81ca-62e226deff03-openshift-service-ca" pod "deaba6ad-8109-48f9-81ca-62e226deff03" (UID: "deaba6ad-8109-48f9-81ca-62e226deff03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:23.833222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.829357    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deaba6ad-8109-48f9-81ca-62e226deff03-kube-api-access-788mm" pod "deaba6ad-8109-48f9-81ca-62e226deff03" (UID: "deaba6ad-8109-48f9-81ca-62e226deff03"). InnerVolumeSpecName "kube-api-access-788mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:23.848466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.843360    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-maas-api-tls" pod "deaba6ad-8109-48f9-81ca-62e226deff03" (UID: "deaba6ad-8109-48f9-81ca-62e226deff03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:23.876228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.873353    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-metrics-certs" pod "deaba6ad-8109-48f9-81ca-62e226deff03" (UID: "deaba6ad-8109-48f9-81ca-62e226deff03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:23.903222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.902586    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/deaba6ad-8109-48f9-81ca-62e226deff03-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:23.903222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.902622    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788mm\" (UniqueName: \"kubernetes.io/projected/deaba6ad-8109-48f9-81ca-62e226deff03-kube-api-access-788mm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:23.903222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.902639    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:23.903222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:23.902652    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/deaba6ad-8109-48f9-81ca-62e226deff03-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.051509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.041078    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:24.263408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.263374    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:20:24.298206 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.298164    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:20:24.405253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.405163    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main" containerID="cri-o://c0dd5d7c0531938c118513b2ddf3bd9c9ec1d233d3d7b97d680653613a400ece" gracePeriod=45
Nov 03 20:20:24.411925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.411900    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4103a6-0edc-448e-b349-7679568bf8d2-kube-api-access-mp24p\" (UniqueName: \"kubernetes.io/projected/7b4103a6-0edc-448e-b349-7679568bf8d2-kube-api-access-mp24p\") pod \"7b4103a6-0edc-448e-b349-7679568bf8d2\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") "
Nov 03 20:20:24.412047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.411991    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358e261d-eb63-493c-bc15-8c202ee4af7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e261d-eb63-493c-bc15-8c202ee4af7f-plugins-config-volume\") pod \"358e261d-eb63-493c-bc15-8c202ee4af7f\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") "
Nov 03 20:20:24.412047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.412029    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358e261d-eb63-493c-bc15-8c202ee4af7f-kube-api-access-svwc6\" (UniqueName: \"kubernetes.io/projected/358e261d-eb63-493c-bc15-8c202ee4af7f-kube-api-access-svwc6\") pod \"358e261d-eb63-493c-bc15-8c202ee4af7f\" (UID: \"358e261d-eb63-493c-bc15-8c202ee4af7f\") "
Nov 03 20:20:24.412186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.412067    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4103a6-0edc-448e-b349-7679568bf8d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4103a6-0edc-448e-b349-7679568bf8d2-plugins-config-volume\") pod \"7b4103a6-0edc-448e-b349-7679568bf8d2\" (UID: \"7b4103a6-0edc-448e-b349-7679568bf8d2\") "
Nov 03 20:20:24.412497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.412464    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358e261d-eb63-493c-bc15-8c202ee4af7f-plugins-config-volume" pod "358e261d-eb63-493c-bc15-8c202ee4af7f" (UID: "358e261d-eb63-493c-bc15-8c202ee4af7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:24.412588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.412537    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4103a6-0edc-448e-b349-7679568bf8d2-plugins-config-volume" pod "7b4103a6-0edc-448e-b349-7679568bf8d2" (UID: "7b4103a6-0edc-448e-b349-7679568bf8d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:24.412794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.412762    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/358e261d-eb63-493c-bc15-8c202ee4af7f-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.412794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.412791    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4103a6-0edc-448e-b349-7679568bf8d2-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.414638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.414617    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4103a6-0edc-448e-b349-7679568bf8d2-kube-api-access-mp24p" pod "7b4103a6-0edc-448e-b349-7679568bf8d2" (UID: "7b4103a6-0edc-448e-b349-7679568bf8d2"). InnerVolumeSpecName "kube-api-access-mp24p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:24.414756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.414647    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358e261d-eb63-493c-bc15-8c202ee4af7f-kube-api-access-svwc6" pod "358e261d-eb63-493c-bc15-8c202ee4af7f" (UID: "358e261d-eb63-493c-bc15-8c202ee4af7f"). InnerVolumeSpecName "kube-api-access-svwc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:24.425302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.425275    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main" containerID="cri-o://b255f3032123e2f76ee13a4cf97b787e46204609b2696f0faa0b0fb08640d8b6" gracePeriod=45
Nov 03 20:20:24.439639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.439609    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main" containerID="cri-o://572f612044adffd4127ba5934144352df16b51f96e938dee6113249c379abe6b" gracePeriod=45
Nov 03 20:20:24.513996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.513968    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwc6\" (UniqueName: \"kubernetes.io/projected/358e261d-eb63-493c-bc15-8c202ee4af7f-kube-api-access-svwc6\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.514110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.513999    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp24p\" (UniqueName: \"kubernetes.io/projected/7b4103a6-0edc-448e-b349-7679568bf8d2-kube-api-access-mp24p\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.533022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.532798    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" podUID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" containerName="payload-processing" containerID="cri-o://709907f46bf9b4f0a46b55e7f130ccc1706b5bc2bf660ca19678c19b615b23b7" gracePeriod=25
Nov 03 20:20:24.541911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.541868    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" podUID="60502a0d-1c95-4527-a002-7b71716a397b" containerName="payload-pre-processing" containerID="cri-o://90de7e0df5d90bc10e398881b372726463a34d5137943892ece604e9f047e691" gracePeriod=25
Nov 03 20:20:24.600711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.600675    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:24.613324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.610628    5282 generic.go:356] "Generic (PLEG): container finished" podID="358e261d-eb63-493c-bc15-8c202ee4af7f" containerID="c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6" exitCode=0
Nov 03 20:20:24.613324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.610744    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" event={"ID":"358e261d-eb63-493c-bc15-8c202ee4af7f","Type":"ContainerDied","Data":"c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6"}
Nov 03 20:20:24.613324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.610793    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4" event={"ID":"358e261d-eb63-493c-bc15-8c202ee4af7f","Type":"ContainerDied","Data":"850d0759fcc845605f7469d0e5eff6f960af8939223b1036b4c6a9f03d5e4aaa"}
Nov 03 20:20:24.613324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.610819    5282 scope.go:122] "RemoveContainer" containerID="c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6"
Nov 03 20:20:24.613324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.610980    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"
Nov 03 20:20:24.754073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.753948    5282 generic.go:356] "Generic (PLEG): container finished" podID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerID="b255f3032123e2f76ee13a4cf97b787e46204609b2696f0faa0b0fb08640d8b6" exitCode=0
Nov 03 20:20:24.754073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.754009    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" event={"ID":"c8a5328f-7423-4d5f-8b53-eb8a10411b75","Type":"ContainerDied","Data":"b255f3032123e2f76ee13a4cf97b787e46204609b2696f0faa0b0fb08640d8b6"}
Nov 03 20:20:24.757964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.756544    5282 generic.go:356] "Generic (PLEG): container finished" podID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerID="572f612044adffd4127ba5934144352df16b51f96e938dee6113249c379abe6b" exitCode=0
Nov 03 20:20:24.758528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.758504    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" event={"ID":"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf","Type":"ContainerDied","Data":"572f612044adffd4127ba5934144352df16b51f96e938dee6113249c379abe6b"}
Nov 03 20:20:24.777026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.776927    5282 scope.go:122] "RemoveContainer" containerID="c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6"
Nov 03 20:20:24.781652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.779229    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"]
Nov 03 20:20:24.785926 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:24.785889    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6\": container with ID starting with c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6 not found: ID does not exist" containerID="c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6"
Nov 03 20:20:24.786039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.785940    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6"} err="failed to get container status \"c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6\": rpc error: code = NotFound desc = could not find container \"c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6\": container with ID starting with c293c17e5ae165acb1b6c62fb0ac3feb92ddf0c7bc4e8c660633c2c4f5f657b6 not found: ID does not exist"
Nov 03 20:20:24.786039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.785964    5282 generic.go:356] "Generic (PLEG): container finished" podID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerID="c0dd5d7c0531938c118513b2ddf3bd9c9ec1d233d3d7b97d680653613a400ece" exitCode=0
Nov 03 20:20:24.786039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.785989    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" event={"ID":"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22","Type":"ContainerDied","Data":"c0dd5d7c0531938c118513b2ddf3bd9c9ec1d233d3d7b97d680653613a400ece"}
Nov 03 20:20:24.786259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.786129    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2092ba-5495d58799-bxvt4"]
Nov 03 20:20:24.788895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.788852    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6" event={"ID":"deaba6ad-8109-48f9-81ca-62e226deff03","Type":"ContainerDied","Data":"edf9547ceb1addd7007254e8422223a1ed37fc4119ee42cb6c578527e334cc0a"}
Nov 03 20:20:24.788992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.788912    5282 scope.go:122] "RemoveContainer" containerID="d1a93542c7e4a9d6031960099b6b96aeae36beacb0c4049cace18211b78d4e9b"
Nov 03 20:20:24.790101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.789839    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"
Nov 03 20:20:24.791389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.791308    5282 generic.go:356] "Generic (PLEG): container finished" podID="7b4103a6-0edc-448e-b349-7679568bf8d2" containerID="ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546" exitCode=0
Nov 03 20:20:24.791389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.791363    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" event={"ID":"7b4103a6-0edc-448e-b349-7679568bf8d2","Type":"ContainerDied","Data":"ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546"}
Nov 03 20:20:24.791524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.791401    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt" event={"ID":"7b4103a6-0edc-448e-b349-7679568bf8d2","Type":"ContainerDied","Data":"21c1914f6d097aa4027294c641cfb7dbbcabd078941fba23b82a493a363119b3"}
Nov 03 20:20:24.791757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.791687    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"
Nov 03 20:20:24.804344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.804322    5282 scope.go:122] "RemoveContainer" containerID="ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546"
Nov 03 20:20:24.829975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.828541    5282 scope.go:122] "RemoveContainer" containerID="ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546"
Nov 03 20:20:24.829975 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:24.829115    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546\": container with ID starting with ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546 not found: ID does not exist" containerID="ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546"
Nov 03 20:20:24.829975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.829158    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546"} err="failed to get container status \"ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546\": rpc error: code = NotFound desc = could not find container \"ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546\": container with ID starting with ead48fc3db1992e9c98e08a27dd0c0d419b1c4c7d68190e811949e112335d546 not found: ID does not exist"
Nov 03 20:20:24.838567 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.836259    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:20:24.844006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.843982    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"]
Nov 03 20:20:24.851249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.851224    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b1aed-5f787b76bc-g6hv6"]
Nov 03 20:20:24.858128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.858104    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:20:24.905855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.905809    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:20:24.928245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.924877    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"]
Nov 03 20:20:24.941664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.941630    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2092ba-645468456d-bstpt"]
Nov 03 20:20:24.951053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951013    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-dshm\") pod \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") "
Nov 03 20:20:24.951254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951068    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tmp-dir\") pod \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") "
Nov 03 20:20:24.951254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951133    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-model-cache\") pod \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") "
Nov 03 20:20:24.951254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951172    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-home\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-home\") pod \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") "
Nov 03 20:20:24.951478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951290    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a5328f-7423-4d5f-8b53-eb8a10411b75-kube-api-access-r7sfb\" (UniqueName: \"kubernetes.io/projected/c8a5328f-7423-4d5f-8b53-eb8a10411b75-kube-api-access-r7sfb\") pod \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") "
Nov 03 20:20:24.951540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951491    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-model-cache\") pod \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") "
Nov 03 20:20:24.951595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951550    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tls-certs\") pod \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") "
Nov 03 20:20:24.951595 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951563    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-model-cache" pod "c8a5328f-7423-4d5f-8b53-eb8a10411b75" (UID: "c8a5328f-7423-4d5f-8b53-eb8a10411b75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.951705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951608    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-home\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-home\") pod \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") "
Nov 03 20:20:24.951705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951673    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tmp-dir\") pod \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") "
Nov 03 20:20:24.951803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951737    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-kube-api-access-9wv62\" (UniqueName: \"kubernetes.io/projected/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-kube-api-access-9wv62\") pod \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") "
Nov 03 20:20:24.951803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951763    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-dshm\") pod \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\" (UID: \"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22\") "
Nov 03 20:20:24.951906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951805    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tls-certs\") pod \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\" (UID: \"c8a5328f-7423-4d5f-8b53-eb8a10411b75\") "
Nov 03 20:20:24.952770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.952608    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.952770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.951761    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tmp-dir" pod "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" (UID: "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.952770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.952342    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-home" pod "c8a5328f-7423-4d5f-8b53-eb8a10411b75" (UID: "c8a5328f-7423-4d5f-8b53-eb8a10411b75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.952770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.952616    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-home" pod "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" (UID: "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.953176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.953122    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tmp-dir" pod "c8a5328f-7423-4d5f-8b53-eb8a10411b75" (UID: "c8a5328f-7423-4d5f-8b53-eb8a10411b75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.953407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.953381    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-model-cache" pod "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" (UID: "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.966254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.965329    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tls-certs" pod "c8a5328f-7423-4d5f-8b53-eb8a10411b75" (UID: "c8a5328f-7423-4d5f-8b53-eb8a10411b75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:24.966254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.965446    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-kube-api-access-9wv62" pod "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" (UID: "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22"). InnerVolumeSpecName "kube-api-access-9wv62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:24.966254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.965500    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-dshm" pod "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" (UID: "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.966254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.965557    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tls-certs" pod "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" (UID: "8c41fa90-ab28-419d-a5a7-6e19f3e1ac22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:24.967375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.967338    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-dshm" pod "c8a5328f-7423-4d5f-8b53-eb8a10411b75" (UID: "c8a5328f-7423-4d5f-8b53-eb8a10411b75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:24.968957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:24.968932    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a5328f-7423-4d5f-8b53-eb8a10411b75-kube-api-access-r7sfb" pod "c8a5328f-7423-4d5f-8b53-eb8a10411b75" (UID: "c8a5328f-7423-4d5f-8b53-eb8a10411b75"). InnerVolumeSpecName "kube-api-access-r7sfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:25.053389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.053295    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-home\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-home\") pod \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") "
Nov 03 20:20:25.053579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.053557    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-kube-api-access-gmnrx\" (UniqueName: \"kubernetes.io/projected/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-kube-api-access-gmnrx\") pod \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") "
Nov 03 20:20:25.053742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.053728    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tls-certs\") pod \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") "
Nov 03 20:20:25.053842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.053826    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-dshm\") pod \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") "
Nov 03 20:20:25.053901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.053870    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-model-cache\") pod \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") "
Nov 03 20:20:25.053958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.053910    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tmp-dir\") pod \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\" (UID: \"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf\") "
Nov 03 20:20:25.054706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.054678    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7sfb\" (UniqueName: \"kubernetes.io/projected/c8a5328f-7423-4d5f-8b53-eb8a10411b75-kube-api-access-r7sfb\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.054878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.054860    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.054977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.054965    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055058    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055150    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055272    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wv62\" (UniqueName: \"kubernetes.io/projected/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-kube-api-access-9wv62\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055369    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055457    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a5328f-7423-4d5f-8b53-eb8a10411b75-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055544    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055640 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055630    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.055720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055709    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a5328f-7423-4d5f-8b53-eb8a10411b75-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.056029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.055996    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tmp-dir" pod "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" (UID: "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:25.056596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.056554    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tls-certs" pod "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" (UID: "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:25.057080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.056915    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-home" pod "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" (UID: "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:25.057080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.057057    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-model-cache" pod "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" (UID: "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:25.059082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.059053    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-kube-api-access-gmnrx" pod "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" (UID: "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf"). InnerVolumeSpecName "kube-api-access-gmnrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:25.060314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.060291    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-dshm" pod "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" (UID: "8e487b72-dd9a-4f64-8292-fdfaf3fbacdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:25.157644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.157608    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.157644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.157644    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmnrx\" (UniqueName: \"kubernetes.io/projected/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-kube-api-access-gmnrx\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.157865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.157660    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.157865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.157677    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.157865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.157692    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.157865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.157705    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.457848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.457814    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358e261d-eb63-493c-bc15-8c202ee4af7f" path="/var/lib/kubelet/pods/358e261d-eb63-493c-bc15-8c202ee4af7f/volumes"
Nov 03 20:20:25.458336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.458320    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4103a6-0edc-448e-b349-7679568bf8d2" path="/var/lib/kubelet/pods/7b4103a6-0edc-448e-b349-7679568bf8d2/volumes"
Nov 03 20:20:25.458751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.458738    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deaba6ad-8109-48f9-81ca-62e226deff03" path="/var/lib/kubelet/pods/deaba6ad-8109-48f9-81ca-62e226deff03/volumes"
Nov 03 20:20:25.591932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.591902    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:25.803587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.803556    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"
Nov 03 20:20:25.803587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.803564    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk" event={"ID":"c8a5328f-7423-4d5f-8b53-eb8a10411b75","Type":"ContainerDied","Data":"b0957725e70e5d205a5c7c47d34d14f7b4dddc9825af38d40ba9e938daf2b355"}
Nov 03 20:20:25.803826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.803610    5282 scope.go:122] "RemoveContainer" containerID="b255f3032123e2f76ee13a4cf97b787e46204609b2696f0faa0b0fb08640d8b6"
Nov 03 20:20:25.804983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.804964    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"
Nov 03 20:20:25.805098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.804998    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm" event={"ID":"8e487b72-dd9a-4f64-8292-fdfaf3fbacdf","Type":"ContainerDied","Data":"32d5b6aa8df9817f9b64370c888d244438e016796c318f8b5d11a56f155ac1de"}
Nov 03 20:20:25.806466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.806447    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"
Nov 03 20:20:25.806652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.806542    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w" event={"ID":"8c41fa90-ab28-419d-a5a7-6e19f3e1ac22","Type":"ContainerDied","Data":"d81f2b63a1ab57c329c9b46b2d4b1bdc23b57d861a798639b4595f14a13d2972"}
Nov 03 20:20:25.815858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.815828    5282 scope.go:122] "RemoveContainer" containerID="572f612044adffd4127ba5934144352df16b51f96e938dee6113249c379abe6b"
Nov 03 20:20:25.826109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.826083    5282 scope.go:122] "RemoveContainer" containerID="c0dd5d7c0531938c118513b2ddf3bd9c9ec1d233d3d7b97d680653613a400ece"
Nov 03 20:20:25.828344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.828318    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"]
Nov 03 20:20:25.832824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.832798    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/premium-w4-5348ca-kserve-7cbdb8c46b-j2jpk"]
Nov 03 20:20:25.852492 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.852458    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"]
Nov 03 20:20:25.856367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.856339    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/simulator-w4-5348ca-kserve-6ff4dc9755-pb8rm"]
Nov 03 20:20:25.860519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.860497    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"]
Nov 03 20:20:25.863923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.863901    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5348ca/distinct-2-w4-5348ca-kserve-6ddf876458-svl5w"]
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.910984    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"]
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912134    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912158    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912182    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912190    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912229    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912236    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912252    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912259    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912280    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912287    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912297    5282 cpu_manager.go:414] "containerMap: removing container" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912306    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912315    5282 cpu_manager.go:414] "containerMap: removing container" podUID="7b4103a6-0edc-448e-b349-7679568bf8d2" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912324    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4103a6-0edc-448e-b349-7679568bf8d2" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912337    5282 cpu_manager.go:414] "containerMap: removing container" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912345    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912359    5282 cpu_manager.go:414] "containerMap: removing container" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912366    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912384    5282 cpu_manager.go:414] "containerMap: removing container" podUID="fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912391    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912408    5282 cpu_manager.go:414] "containerMap: removing container" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912415    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912428    5282 cpu_manager.go:414] "containerMap: removing container" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912436    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912449    5282 cpu_manager.go:414] "containerMap: removing container" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912456    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912469    5282 cpu_manager.go:414] "containerMap: removing container" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912476    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912497    5282 cpu_manager.go:414] "containerMap: removing container" podUID="af36f373-ad1c-474c-845e-95586c8304ef" containerName="maas-api"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912506    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af36f373-ad1c-474c-845e-95586c8304ef" containerName="maas-api"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912528    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912535    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912546    5282 cpu_manager.go:414] "containerMap: removing container" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912553    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912562    5282 cpu_manager.go:414] "containerMap: removing container" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912568    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912582    5282 cpu_manager.go:414] "containerMap: removing container" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912589    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912603    5282 cpu_manager.go:414] "containerMap: removing container" podUID="deaba6ad-8109-48f9-81ca-62e226deff03" containerName="maas-api"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912609    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deaba6ad-8109-48f9-81ca-62e226deff03" containerName="maas-api"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912624    5282 cpu_manager.go:414] "containerMap: removing container" podUID="477c28ea-7bed-4489-9bbc-5c0b812b317e" containerName="payload-pre-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912631    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477c28ea-7bed-4489-9bbc-5c0b812b317e" containerName="payload-pre-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912649    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ae51af71-0640-4d6d-bec2-f819ef9ebad6" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912656    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae51af71-0640-4d6d-bec2-f819ef9ebad6" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912668    5282 cpu_manager.go:414] "containerMap: removing container" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912675    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912690    5282 cpu_manager.go:414] "containerMap: removing container" podUID="358e261d-eb63-493c-bc15-8c202ee4af7f" containerName="payload-pre-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912697    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358e261d-eb63-493c-bc15-8c202ee4af7f" containerName="payload-pre-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912868    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="67151051-1ed2-4a57-a34c-4465e6538648" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912885    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c2b3db-5451-4b5a-8313-9fc3fb58db6b" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912895    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912905    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912915    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="deaba6ad-8109-48f9-81ca-62e226deff03" containerName="maas-api"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912927    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="477c28ea-7bed-4489-9bbc-5c0b812b317e" containerName="payload-pre-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912938    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae51af71-0640-4d6d-bec2-f819ef9ebad6" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912948    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f48c89-4498-48ff-8c32-4a2c796151f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912957    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="358e261d-eb63-493c-bc15-8c202ee4af7f" containerName="payload-pre-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912968    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.912977    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b558ab4-c66b-48b8-a93d-c8f9a48211dd" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913038    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913050    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb363fe-2c71-4a71-8a21-9b366a5d42a5" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913062    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913073    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913082    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd7bc8d-1d27-4642-a1fb-676c08dae7bc" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913089    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4103a6-0edc-448e-b349-7679568bf8d2" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913098    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8b4b8d-74e1-4c86-b79e-e9c9b76fccfe" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913106    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913114    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa329b0a-2fe4-4229-bbde-c5dc6a0ebcea" containerName="payload-processing"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913123    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" containerName="main"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913133    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="be91e70c-072f-4fab-be70-d26878da83e0" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913142    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913150    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="60336029-d74c-4227-a77f-7f2bfc2a17f1" containerName="istio-proxy"
Nov 03 20:20:25.914243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.913160    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="af36f373-ad1c-474c-845e-95586c8304ef" containerName="maas-api"
Nov 03 20:20:25.925833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.920944    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:25.925833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.925683    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"]
Nov 03 20:20:25.928008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.927348    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22a567\"/\"kube-root-ca.crt\""
Nov 03 20:20:25.928008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.927464    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22a567\"/\"default-dockercfg-2hvfb\""
Nov 03 20:20:25.928008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.927978    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-22a567\"/\"openshift-service-ca.crt\""
Nov 03 20:20:25.928297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.928272    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-22a567\"/\"test-model-22a567-kserve-self-signed-certs\""
Nov 03 20:20:25.996567 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:25.996534    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:26.088319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.070682    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-model-cache\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.088319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.070851    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tmp-dir\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.088319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.070898    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-dshm\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.088319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.070944    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tls-certs\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.088319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.071014    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hht\" (UniqueName: \"kubernetes.io/projected/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-kube-api-access-b5hht\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.088319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.071161    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-home\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.175222    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-home\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.175362    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-model-cache\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.175509    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tmp-dir\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.175547    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-dshm\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.175593    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tls-certs\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.175680    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hht\" (UniqueName: \"kubernetes.io/projected/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-kube-api-access-b5hht\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.176571    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-home\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.176841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.176797    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-model-cache\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.189675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.177028    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tmp-dir\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.196962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.196880    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tls-certs\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.197444 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.197394    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-dshm\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.227941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.223392    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hht\" (UniqueName: \"kubernetes.io/projected/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-kube-api-access-b5hht\") pod \"test-model-22a567-kserve-6d86688d85-pcrpr\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") " pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.243777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.243495    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:26.417954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.417869    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:26.556307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.556260    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"]
Nov 03 20:20:26.557157 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:20:26.557133    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7d8c32f_89d6_403e_ac8f_47fcf85f2632.slice/crio-643592726d9f1cfd60e6731e1740d4362056548a9ba2120ec7580a884f057c66 WatchSource:0}: Error finding container 643592726d9f1cfd60e6731e1740d4362056548a9ba2120ec7580a884f057c66: Status 404 returned error can't find the container with id 643592726d9f1cfd60e6731e1740d4362056548a9ba2120ec7580a884f057c66
Nov 03 20:20:26.640296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.640256    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" podUID="030e2c8d-47b1-4787-9efb-393780359cc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:26.826904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.826845    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" event={"ID":"c7d8c32f-89d6-403e-ac8f-47fcf85f2632","Type":"ContainerStarted","Data":"fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557"}
Nov 03 20:20:26.827086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.826912    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" event={"ID":"c7d8c32f-89d6-403e-ac8f-47fcf85f2632","Type":"ContainerStarted","Data":"643592726d9f1cfd60e6731e1740d4362056548a9ba2120ec7580a884f057c66"}
Nov 03 20:20:26.849512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:26.849453    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" podStartSLOduration=1.849437191 podStartE2EDuration="1.849437191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:26.84648857 +0000 UTC m=+2419.515690039" watchObservedRunningTime="2026-11-03 20:20:26.849437191 +0000 UTC m=+2419.518638702"
Nov 03 20:20:27.244920 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.244891    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:27.297400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.297288    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:27.460723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.460685    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c41fa90-ab28-419d-a5a7-6e19f3e1ac22" path="/var/lib/kubelet/pods/8c41fa90-ab28-419d-a5a7-6e19f3e1ac22/volumes"
Nov 03 20:20:27.461227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.461180    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e487b72-dd9a-4f64-8292-fdfaf3fbacdf" path="/var/lib/kubelet/pods/8e487b72-dd9a-4f64-8292-fdfaf3fbacdf/volumes"
Nov 03 20:20:27.461670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.461653    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a5328f-7423-4d5f-8b53-eb8a10411b75" path="/var/lib/kubelet/pods/c8a5328f-7423-4d5f-8b53-eb8a10411b75/volumes"
Nov 03 20:20:27.463314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.463292    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:20:27.635086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.635053    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030e2c8d-47b1-4787-9efb-393780359cc1-kube-api-access-5rsvj\" (UniqueName: \"kubernetes.io/projected/030e2c8d-47b1-4787-9efb-393780359cc1-kube-api-access-5rsvj\") pod \"030e2c8d-47b1-4787-9efb-393780359cc1\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") "
Nov 03 20:20:27.635254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.635168    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030e2c8d-47b1-4787-9efb-393780359cc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e2c8d-47b1-4787-9efb-393780359cc1-plugins-config-volume\") pod \"030e2c8d-47b1-4787-9efb-393780359cc1\" (UID: \"030e2c8d-47b1-4787-9efb-393780359cc1\") "
Nov 03 20:20:27.635542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.635509    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030e2c8d-47b1-4787-9efb-393780359cc1-plugins-config-volume" pod "030e2c8d-47b1-4787-9efb-393780359cc1" (UID: "030e2c8d-47b1-4787-9efb-393780359cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:27.636026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.635898    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030e2c8d-47b1-4787-9efb-393780359cc1-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:27.637407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.637383    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030e2c8d-47b1-4787-9efb-393780359cc1-kube-api-access-5rsvj" pod "030e2c8d-47b1-4787-9efb-393780359cc1" (UID: "030e2c8d-47b1-4787-9efb-393780359cc1"). InnerVolumeSpecName "kube-api-access-5rsvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:27.737281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.737250    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rsvj\" (UniqueName: \"kubernetes.io/projected/030e2c8d-47b1-4787-9efb-393780359cc1-kube-api-access-5rsvj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:27.835925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.835889    5282 generic.go:356] "Generic (PLEG): container finished" podID="030e2c8d-47b1-4787-9efb-393780359cc1" containerID="34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c" exitCode=0
Nov 03 20:20:27.836120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.835942    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"
Nov 03 20:20:27.836120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.835983    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" event={"ID":"030e2c8d-47b1-4787-9efb-393780359cc1","Type":"ContainerDied","Data":"34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c"}
Nov 03 20:20:27.836120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.836038    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7" event={"ID":"030e2c8d-47b1-4787-9efb-393780359cc1","Type":"ContainerDied","Data":"ff5938afab72419321961539ca72157a40354f58f7afc9768310b38155bd7c4e"}
Nov 03 20:20:27.836120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.836065    5282 scope.go:122] "RemoveContainer" containerID="34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c"
Nov 03 20:20:27.847434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.847417    5282 scope.go:122] "RemoveContainer" containerID="34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c"
Nov 03 20:20:27.847751 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:27.847729    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c\": container with ID starting with 34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c not found: ID does not exist" containerID="34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c"
Nov 03 20:20:27.847858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.847760    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c"} err="failed to get container status \"34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c\": rpc error: code = NotFound desc = could not find container \"34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c\": container with ID starting with 34beeadc3b96d408a230529e03f8e34258868f65aa0effa09707684aee09e46c not found: ID does not exist"
Nov 03 20:20:27.862440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.862418    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"]
Nov 03 20:20:27.869064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:27.869038    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-663013-56888c558c-w6kx7"]
Nov 03 20:20:28.097425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:28.097345    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:28.803921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:28.803879    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:29.458045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:29.458010    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030e2c8d-47b1-4787-9efb-393780359cc1" path="/var/lib/kubelet/pods/030e2c8d-47b1-4787-9efb-393780359cc1/volumes"
Nov 03 20:20:29.585827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:29.585768    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:29.715259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:29.715131    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:29.970271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:29.933752    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" podUID="60502a0d-1c95-4527-a002-7b71716a397b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:29.970271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:29.934618    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" podUID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:30.195209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:30.195171    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:30.352673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:30.352625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:30.526508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:30.526472    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:30.678307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:30.676864    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:31.282210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.282173    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:31.402806 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.402755    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:31.626520 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:31.626419    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:20:31.636986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.636949    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"]
Nov 03 20:20:31.638397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638343    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy"
Nov 03 20:20:31.638397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638372    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ff4d7f-c41a-4bd0-ba8a-6a92e13495e9" containerName="istio-proxy"
Nov 03 20:20:31.638602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638434    5282 cpu_manager.go:414] "containerMap: removing container" podUID="030e2c8d-47b1-4787-9efb-393780359cc1" containerName="payload-pre-processing"
Nov 03 20:20:31.638602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638444    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030e2c8d-47b1-4787-9efb-393780359cc1" containerName="payload-pre-processing"
Nov 03 20:20:31.638602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638473    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy"
Nov 03 20:20:31.638602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638482    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy"
Nov 03 20:20:31.638766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638701    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="030e2c8d-47b1-4787-9efb-393780359cc1" containerName="payload-pre-processing"
Nov 03 20:20:31.638766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.638730    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f42d2e4-8389-474b-9c77-eb1fd340edad" containerName="istio-proxy"
Nov 03 20:20:31.645010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.644982    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"]
Nov 03 20:20:31.645151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.645123    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.657177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.657148    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-9ff9f5-openshift-default-dockercfg-6tzbj\""
Nov 03 20:20:31.690181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.690149    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.779783    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-data\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.779849    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-token\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.779919    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk28b\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-kube-api-access-qk28b\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.779982    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-credential-socket\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.780017    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-envoy\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.780046    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-certs\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.780069    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-podinfo\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.780169    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-socket\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.783221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.780270    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9f6315-7999-43a5-8ca3-6434b1058d58-istiod-ca-cert\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.889598    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-credential-socket\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.889666    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-envoy\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.889700    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-certs\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.889726    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-podinfo\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.889848    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-socket\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.889941    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9f6315-7999-43a5-8ca3-6434b1058d58-istiod-ca-cert\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.890071    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-data\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.890115    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-token\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.890287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.890215    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk28b\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-kube-api-access-qk28b\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.895229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.891011    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-credential-socket\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.895229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.891733    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-socket\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.895229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.891991    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-certs\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.895229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.892643    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-data\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.895229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.893251    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9f6315-7999-43a5-8ca3-6434b1058d58-istiod-ca-cert\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.939221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.931459    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-podinfo\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.939221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.931701    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-envoy\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:31.986223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:31.982373    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-token\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:32.024225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.008151    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk28b\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-kube-api-access-qk28b\") pod \"e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") " pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:32.024225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.020649    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.179281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.166857    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.277236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.264685    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:32.305228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.301098    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.433301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.432108    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.694277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.690363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:32.768645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.768609    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"]
Nov 03 20:20:32.768991 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:20:32.768969    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9f6315_7999_43a5_8ca3_6434b1058d58.slice/crio-be84b97f80fe3c074b0a3793628f7096de171481e2b0ea02981a8a05775bdf15 WatchSource:0}: Error finding container be84b97f80fe3c074b0a3793628f7096de171481e2b0ea02981a8a05775bdf15: Status 404 returned error can't find the container with id be84b97f80fe3c074b0a3793628f7096de171481e2b0ea02981a8a05775bdf15
Nov 03 20:20:32.771798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.771765    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:32.771879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.771830    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:32.771879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.771858    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:32.890264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:32.890229    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:33.075804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:33.075764    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" event={"ID":"4f9f6315-7999-43a5-8ca3-6434b1058d58","Type":"ContainerStarted","Data":"046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6"}
Nov 03 20:20:33.075989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:33.075831    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" event={"ID":"4f9f6315-7999-43a5-8ca3-6434b1058d58","Type":"ContainerStarted","Data":"be84b97f80fe3c074b0a3793628f7096de171481e2b0ea02981a8a05775bdf15"}
Nov 03 20:20:33.097219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:33.097143    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podStartSLOduration=2.097129712 podStartE2EDuration="2.097129712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:33.094725252 +0000 UTC m=+2425.763926722" watchObservedRunningTime="2026-11-03 20:20:33.097129712 +0000 UTC m=+2425.766331181"
Nov 03 20:20:33.283220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:33.272088    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:33.289524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:33.288520    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:33.547976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:33.547931    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:34.093946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:34.093914    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:34.094846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:34.094825    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:20:34.650932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:34.650892    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:35.566047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:35.566000    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:35.824284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:35.818901    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:35.972937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:35.972866    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.121829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:36.121741    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.304270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:36.284383    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:36.525986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:36.525946    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:36.636261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:36.621500    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:37.232229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.232059    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:37.270225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.269897    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"]
Nov 03 20:20:37.364235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.362415    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"]
Nov 03 20:20:37.364235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.362542    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:37.398298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.380629    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-9ff9f5-dockercfg-clczf\""
Nov 03 20:20:37.489302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.486870    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-9ff9f5\""
Nov 03 20:20:37.514230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.512705    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:37.535457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.531885    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"]
Nov 03 20:20:37.674632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.674585    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlqw2\" (UniqueName: \"kubernetes.io/projected/bf9a923f-9f14-4691-89a9-5724b7e49c9b-kube-api-access-tlqw2\") pod \"payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:37.675155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.675121    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9a923f-9f14-4691-89a9-5724b7e49c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:37.788290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.779427    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlqw2\" (UniqueName: \"kubernetes.io/projected/bf9a923f-9f14-4691-89a9-5724b7e49c9b-kube-api-access-tlqw2\") pod \"payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:37.793498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.790478    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9a923f-9f14-4691-89a9-5724b7e49c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:37.837223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.793747    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9a923f-9f14-4691-89a9-5724b7e49c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:37.837223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.815425    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:37.837223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.833014    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlqw2\" (UniqueName: \"kubernetes.io/projected/bf9a923f-9f14-4691-89a9-5724b7e49c9b-kube-api-access-tlqw2\") pod \"payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:37.893959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.893924    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"]
Nov 03 20:20:37.913541 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.912993    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:37.919685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.919139    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhxs\" (UniqueName: \"kubernetes.io/projected/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-kube-api-access-tkhxs\") pod \"payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:37.977507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:37.977474    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:38.051270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.041834    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:38.051270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.041946    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhxs\" (UniqueName: \"kubernetes.io/projected/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-kube-api-access-tkhxs\") pod \"payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:38.051270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.045471    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:38.051270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.049709    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:20:38.074252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.072015    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhxs\" (UniqueName: \"kubernetes.io/projected/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-kube-api-access-tkhxs\") pod \"payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:38.373293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.371372    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:38.413227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.395643    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:38.735068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.735036    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"]
Nov 03 20:20:38.931223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:38.929658    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"]
Nov 03 20:20:39.023839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.023810    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:39.648232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.648165    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r" event={"ID":"4cbdbcd5-4b29-425c-a0af-e0165833c2c1","Type":"ContainerStarted","Data":"59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633"}
Nov 03 20:20:39.648780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.648759    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r" event={"ID":"4cbdbcd5-4b29-425c-a0af-e0165833c2c1","Type":"ContainerStarted","Data":"0306838fd85ae5ff603adc43d012e35d4dde846b4c734722039726e7595a20a5"}
Nov 03 20:20:39.648911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.648898    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:39.650225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.650190    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65" event={"ID":"bf9a923f-9f14-4691-89a9-5724b7e49c9b","Type":"ContainerStarted","Data":"e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b"}
Nov 03 20:20:39.650349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.650283    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65" event={"ID":"bf9a923f-9f14-4691-89a9-5724b7e49c9b","Type":"ContainerStarted","Data":"093f31ef8069480e423822fd54414017fd6ef858d2f5df65efbb7a6341102312"}
Nov 03 20:20:39.650349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.650317    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:39.710217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.709661    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r" podStartSLOduration=2.709641981 podStartE2EDuration="2.709641981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:39.696162546 +0000 UTC m=+2432.365364019" watchObservedRunningTime="2026-11-03 20:20:39.709641981 +0000 UTC m=+2432.378843452"
Nov 03 20:20:39.731510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.731436    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65" podStartSLOduration=2.731420932 podStartE2EDuration="2.731420932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:39.726595915 +0000 UTC m=+2432.395797386" watchObservedRunningTime="2026-11-03 20:20:39.731420932 +0000 UTC m=+2432.400622405"
Nov 03 20:20:39.935269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.930009    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:39.935269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.932327    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" podUID="60502a0d-1c95-4527-a002-7b71716a397b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:39.940845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:39.940802    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" podUID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:20:40.161627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:40.161589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:40.320727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:40.319251    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:40.452367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:40.452332    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"]
Nov 03 20:20:40.452854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:40.452823    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy" containerID="cri-o://f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54" gracePeriod=30
Nov 03 20:20:40.524033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:40.523906    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:40.934218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:40.922269    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:41.599780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:41.599687    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:42.225284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.225246    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:42.424031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.423996    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:42.564501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.564461    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:42.727262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.716462    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"]
Nov 03 20:20:42.727558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.727534    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"]
Nov 03 20:20:42.727845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.727821    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" containerID="cri-o://c9283a2bc3eb8cabacbe72d21ee521d2ba00c6c87ffd0c5de27a9b57c4020b3d" gracePeriod=30
Nov 03 20:20:42.728425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.728398    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy" containerID="cri-o://23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f" gracePeriod=30
Nov 03 20:20:42.794746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:42.794645    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:43.119518 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:43.119352    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:43.280671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:43.273215    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:43.542986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:43.542949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:44.117228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.115546    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:44.260932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.260831    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:44.398144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.398054    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:44.763475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.763441    5282 generic.go:356] "Generic (PLEG): container finished" podID="60502a0d-1c95-4527-a002-7b71716a397b" containerID="90de7e0df5d90bc10e398881b372726463a34d5137943892ece604e9f047e691" exitCode=0
Nov 03 20:20:44.763869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.763707    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" event={"ID":"60502a0d-1c95-4527-a002-7b71716a397b","Type":"ContainerDied","Data":"90de7e0df5d90bc10e398881b372726463a34d5137943892ece604e9f047e691"}
Nov 03 20:20:44.766315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.766277    5282 generic.go:356] "Generic (PLEG): container finished" podID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" containerID="709907f46bf9b4f0a46b55e7f130ccc1706b5bc2bf660ca19678c19b615b23b7" exitCode=0
Nov 03 20:20:44.766454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.766339    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" event={"ID":"bfd9726a-f255-4fe9-b09b-b1aa20027e09","Type":"ContainerDied","Data":"709907f46bf9b4f0a46b55e7f130ccc1706b5bc2bf660ca19678c19b615b23b7"}
Nov 03 20:20:44.931728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.931306    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:20:44.945239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:44.940118    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:20:45.030765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.030714    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd9726a-f255-4fe9-b09b-b1aa20027e09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd9726a-f255-4fe9-b09b-b1aa20027e09-plugins-config-volume\") pod \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") "
Nov 03 20:20:45.030976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.030817    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd9726a-f255-4fe9-b09b-b1aa20027e09-kube-api-access-clfvq\" (UniqueName: \"kubernetes.io/projected/bfd9726a-f255-4fe9-b09b-b1aa20027e09-kube-api-access-clfvq\") pod \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\" (UID: \"bfd9726a-f255-4fe9-b09b-b1aa20027e09\") "
Nov 03 20:20:45.031951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.031919    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd9726a-f255-4fe9-b09b-b1aa20027e09-plugins-config-volume" pod "bfd9726a-f255-4fe9-b09b-b1aa20027e09" (UID: "bfd9726a-f255-4fe9-b09b-b1aa20027e09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:45.032501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.032473    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60502a0d-1c95-4527-a002-7b71716a397b-plugins-config-volume" pod "60502a0d-1c95-4527-a002-7b71716a397b" (UID: "60502a0d-1c95-4527-a002-7b71716a397b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:45.032606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.031093    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60502a0d-1c95-4527-a002-7b71716a397b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60502a0d-1c95-4527-a002-7b71716a397b-plugins-config-volume\") pod \"60502a0d-1c95-4527-a002-7b71716a397b\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") "
Nov 03 20:20:45.032677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.032630    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60502a0d-1c95-4527-a002-7b71716a397b-kube-api-access-4mbcq\" (UniqueName: \"kubernetes.io/projected/60502a0d-1c95-4527-a002-7b71716a397b-kube-api-access-4mbcq\") pod \"60502a0d-1c95-4527-a002-7b71716a397b\" (UID: \"60502a0d-1c95-4527-a002-7b71716a397b\") "
Nov 03 20:20:45.033465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.033439    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd9726a-f255-4fe9-b09b-b1aa20027e09-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.033465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.033467    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60502a0d-1c95-4527-a002-7b71716a397b-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.035098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.035073    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd9726a-f255-4fe9-b09b-b1aa20027e09-kube-api-access-clfvq" pod "bfd9726a-f255-4fe9-b09b-b1aa20027e09" (UID: "bfd9726a-f255-4fe9-b09b-b1aa20027e09"). InnerVolumeSpecName "kube-api-access-clfvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:45.037178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.037149    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60502a0d-1c95-4527-a002-7b71716a397b-kube-api-access-4mbcq" pod "60502a0d-1c95-4527-a002-7b71716a397b" (UID: "60502a0d-1c95-4527-a002-7b71716a397b"). InnerVolumeSpecName "kube-api-access-4mbcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:45.041151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.041120    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:45.138185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.137911    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbcq\" (UniqueName: \"kubernetes.io/projected/60502a0d-1c95-4527-a002-7b71716a397b-kube-api-access-4mbcq\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.138185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.137947    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfvq\" (UniqueName: \"kubernetes.io/projected/bfd9726a-f255-4fe9-b09b-b1aa20027e09-kube-api-access-clfvq\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.711868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.711828    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:45.772967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.772936    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:20:45.794033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.793995    5282 generic.go:356] "Generic (PLEG): container finished" podID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerID="f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54" exitCode=0
Nov 03 20:20:45.794239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.794077    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" event={"ID":"9f2ee665-2396-4632-b5eb-8ee520151e07","Type":"ContainerDied","Data":"f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54"}
Nov 03 20:20:45.794239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.794144    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf" event={"ID":"9f2ee665-2396-4632-b5eb-8ee520151e07","Type":"ContainerDied","Data":"5cfcd4e9b4919af77055f8322df0e31b98e0e772b8c72669a43253b6e39c8d43"}
Nov 03 20:20:45.794239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.794177    5282 scope.go:122] "RemoveContainer" containerID="f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54"
Nov 03 20:20:45.794637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.794613    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"
Nov 03 20:20:45.822337 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.803402    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5" event={"ID":"bfd9726a-f255-4fe9-b09b-b1aa20027e09","Type":"ContainerDied","Data":"138cdf38efbba7ca53b8553ff5b43d34335b849d6e7c9cfd8047cb7b5d7745a3"}
Nov 03 20:20:45.836328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.822860    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"
Nov 03 20:20:45.854586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.853990    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-data\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.854586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854186    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-podinfo\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.854586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854287    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-kube-api-access-fx6m4\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-kube-api-access-fx6m4\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.854586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854337    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-credential-socket\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.854586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854506    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-envoy\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.855007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854703    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-certs\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.855007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854744    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-token\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-token\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.855007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854780    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2ee665-2396-4632-b5eb-8ee520151e07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2ee665-2396-4632-b5eb-8ee520151e07-istiod-ca-cert\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.855007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.854832    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-socket\") pod \"9f2ee665-2396-4632-b5eb-8ee520151e07\" (UID: \"9f2ee665-2396-4632-b5eb-8ee520151e07\") "
Nov 03 20:20:45.860862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.855899    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-socket" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.860862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.856971    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2ee665-2396-4632-b5eb-8ee520151e07-istiod-ca-cert" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:45.939700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.936408    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-certs" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.939700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.937690    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-credential-socket" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.939700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.937774    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-data" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:45.978393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.970361    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.978393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.970414    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.978393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.970435    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.978393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.970472    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f2ee665-2396-4632-b5eb-8ee520151e07-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.978393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.970498    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:45.985399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.984958    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd" event={"ID":"60502a0d-1c95-4527-a002-7b71716a397b","Type":"ContainerDied","Data":"19ee746af4cf3f06a06df277996668e67bb5b5a7da009641174499cc52cc1b3a"}
Nov 03 20:20:45.985399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:45.985150    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"
Nov 03 20:20:46.012713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.012611    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-podinfo" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:46.013297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.013240    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-envoy" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:46.020816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.014502    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-kube-api-access-fx6m4" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "kube-api-access-fx6m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:46.020816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.017878    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-token" pod "9f2ee665-2396-4632-b5eb-8ee520151e07" (UID: "9f2ee665-2396-4632-b5eb-8ee520151e07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:46.036369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.036304    5282 scope.go:122] "RemoveContainer" containerID="f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54"
Nov 03 20:20:46.036369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.036363    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"]
Nov 03 20:20:46.042050 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:46.038631    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54\": container with ID starting with f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54 not found: ID does not exist" containerID="f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54"
Nov 03 20:20:46.042050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.038735    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54"} err="failed to get container status \"f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54\": rpc error: code = NotFound desc = could not find container \"f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54\": container with ID starting with f45477c30868aa0ba894f8cc61d2c47ad5e7397e2d36301170ae8a4a25758c54 not found: ID does not exist"
Nov 03 20:20:46.042050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.038782    5282 scope.go:122] "RemoveContainer" containerID="709907f46bf9b4f0a46b55e7f130ccc1706b5bc2bf660ca19678c19b615b23b7"
Nov 03 20:20:46.045298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.045250    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9b1aed-56586c9b6-rrvz5"]
Nov 03 20:20:46.051043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.051007    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"]
Nov 03 20:20:46.074143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.074044    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.074143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.074084    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.074143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.074101    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx6m4\" (UniqueName: \"kubernetes.io/projected/9f2ee665-2396-4632-b5eb-8ee520151e07-kube-api-access-fx6m4\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.074143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.074117    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f2ee665-2396-4632-b5eb-8ee520151e07-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.074731 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.074647    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9b1aed-df66ffd87-rvlcd"]
Nov 03 20:20:46.080019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.079900    5282 scope.go:122] "RemoveContainer" containerID="90de7e0df5d90bc10e398881b372726463a34d5137943892ece604e9f047e691"
Nov 03 20:20:46.135776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.135745    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:46.154333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.154302    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"]
Nov 03 20:20:46.161977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.161946    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5348ca-openshift-default-5b8f4c95d5-5t4mf"]
Nov 03 20:20:46.250158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.250118    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:46.544388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.544350    5282 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:46.544575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.544415    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:46.659054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:46.650868    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:47.340485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.340408    5282 patch_prober.go:29] interesting pod/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:20:47.341062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.340538    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:20:47.361624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.361592    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:47.499067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.499009    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60502a0d-1c95-4527-a002-7b71716a397b" path="/var/lib/kubelet/pods/60502a0d-1c95-4527-a002-7b71716a397b/volumes"
Nov 03 20:20:47.499974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.499949    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" path="/var/lib/kubelet/pods/9f2ee665-2396-4632-b5eb-8ee520151e07/volumes"
Nov 03 20:20:47.500696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.500675    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" path="/var/lib/kubelet/pods/bfd9726a-f255-4fe9-b09b-b1aa20027e09/volumes"
Nov 03 20:20:47.541850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.541793    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:47.976727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:47.976610    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:20:48.014019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.013991    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-credential-socket\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014044    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-token\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-token\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014083    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-podinfo\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014112    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-data\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014163    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-envoy\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014311    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istiod-ca-cert\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014364    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-socket\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014559    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-kube-api-access-84qm4\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-kube-api-access-84qm4\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.014645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.014630    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-certs\") pod \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\" (UID: \"9b5fa49b-6181-408e-87a3-3a63ae7b4a74\") "
Nov 03 20:20:48.015535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.015509    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-certs" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.015703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.015625    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-data" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.015866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.015839    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-credential-socket" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.017166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.017142    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-socket" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.017639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.017618    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istiod-ca-cert" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:48.018604 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.018566    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-envoy" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.020000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.019972    5282 generic.go:356] "Generic (PLEG): container finished" podID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerID="c9283a2bc3eb8cabacbe72d21ee521d2ba00c6c87ffd0c5de27a9b57c4020b3d" exitCode=0
Nov 03 20:20:48.020110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.020076    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" event={"ID":"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e","Type":"ContainerDied","Data":"c9283a2bc3eb8cabacbe72d21ee521d2ba00c6c87ffd0c5de27a9b57c4020b3d"}
Nov 03 20:20:48.020178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.020139    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz" event={"ID":"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e","Type":"ContainerDied","Data":"e48729caceecdf949843bc95b6eb857faa4853d9facfc4677f8d167bdbc8bddd"}
Nov 03 20:20:48.020178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.020169    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48729caceecdf949843bc95b6eb857faa4853d9facfc4677f8d167bdbc8bddd"
Nov 03 20:20:48.021118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.020188    5282 scope.go:122] "RemoveContainer" containerID="552f103a81a916d81f0053af9578427b1fb978148383d30d94b8bdc65941f452"
Nov 03 20:20:48.021118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.020953    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-podinfo" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:48.022025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.021999    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-kube-api-access-84qm4" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "kube-api-access-84qm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:48.023755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.023709    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-token" pod "9b5fa49b-6181-408e-87a3-3a63ae7b4a74" (UID: "9b5fa49b-6181-408e-87a3-3a63ae7b4a74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:48.023887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.023767    5282 generic.go:356] "Generic (PLEG): container finished" podID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerID="ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023" exitCode=0
Nov 03 20:20:48.024084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.023790    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" event={"ID":"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8","Type":"ContainerDied","Data":"ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023"}
Nov 03 20:20:48.024760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.024722    5282 scope.go:122] "RemoveContainer" containerID="ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023"
Nov 03 20:20:48.027049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.026903    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:48.027049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.026999    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:48.027222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.027051    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:48.029074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.028346    5282 generic.go:356] "Generic (PLEG): container finished" podID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerID="23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f" exitCode=0
Nov 03 20:20:48.029074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.028445    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" event={"ID":"9b5fa49b-6181-408e-87a3-3a63ae7b4a74","Type":"ContainerDied","Data":"23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f"}
Nov 03 20:20:48.029074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.028487    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6" event={"ID":"9b5fa49b-6181-408e-87a3-3a63ae7b4a74","Type":"ContainerDied","Data":"fa6335a910a5cb2c4b66d36315ebc2ee12f7c2a1ce17dae16478dffdcf23fc2c"}
Nov 03 20:20:48.029074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.028555    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"
Nov 03 20:20:48.035399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.035373    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:20:48.036696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.036671    5282 scope.go:122] "RemoveContainer" containerID="23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f"
Nov 03 20:20:48.061858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.061695    5282 scope.go:122] "RemoveContainer" containerID="23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f"
Nov 03 20:20:48.065971 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:20:48.065876    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f\": container with ID starting with 23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f not found: ID does not exist" containerID="23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f"
Nov 03 20:20:48.065971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.065920    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f"} err="failed to get container status \"23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f\": rpc error: code = NotFound desc = could not find container \"23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f\": container with ID starting with 23a06d00a8b3c696ec0eaa96f7b41fd5e2735053504a8201b3a10d10defc974f not found: ID does not exist"
Nov 03 20:20:48.067479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.067458    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"]
Nov 03 20:20:48.072923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.072868    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-c68l6"]
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116189    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-socket\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116264    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-token\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-token\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116338    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-certs\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116393    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-data\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116447    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-credential-socket\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116561    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-envoy\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116654    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-kube-api-access-r6v2l\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-kube-api-access-r6v2l\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116768    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-podinfo\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.117430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.116851    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istiod-ca-cert\") pod \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\" (UID: \"a2a8e3d3-c724-43cc-8d27-c5e49d16f85e\") "
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117622    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84qm4\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-kube-api-access-84qm4\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117647    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117663    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117678    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117693    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117707    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117721    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117737    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.117753    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b5fa49b-6181-408e-87a3-3a63ae7b4a74-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.118523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.118287    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-socket" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.119426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.119403    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-data" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.119597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.119579    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-certs" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.120929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.120905    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istiod-ca-cert" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:48.121151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.121130    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-credential-socket" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.126739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.126695    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-podinfo" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:48.126934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.126909    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-envoy" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:48.127031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.126998    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-token" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:48.128462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.128417    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-kube-api-access-r6v2l" pod "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" (UID: "a2a8e3d3-c724-43cc-8d27-c5e49d16f85e"). InnerVolumeSpecName "kube-api-access-r6v2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:48.190150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.190110    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219315    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6v2l\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-kube-api-access-r6v2l\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219356    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219375    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219389    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219403    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219416    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219432    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219445    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.220421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.219457    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:48.490272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.490231    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:48.633121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:48.633083    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:49.048256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.048132    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"
Nov 03 20:20:49.050476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.050442    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" event={"ID":"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8","Type":"ContainerStarted","Data":"aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d"}
Nov 03 20:20:49.052260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.052231    5282 generic.go:356] "Generic (PLEG): container finished" podID="347cb059-324b-4249-b74b-2a18fd8d045a" containerID="5bcedfd738e5b4906ce66fa796b322c0b81482e249ec88e229de9c87bca2b1ec" exitCode=0
Nov 03 20:20:49.052393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.052263    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" event={"ID":"347cb059-324b-4249-b74b-2a18fd8d045a","Type":"ContainerDied","Data":"5bcedfd738e5b4906ce66fa796b322c0b81482e249ec88e229de9c87bca2b1ec"}
Nov 03 20:20:49.052667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.052649    5282 scope.go:122] "RemoveContainer" containerID="5bcedfd738e5b4906ce66fa796b322c0b81482e249ec88e229de9c87bca2b1ec"
Nov 03 20:20:49.054542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.054506    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:49.054638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.054593    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:49.054689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.054636    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:49.156857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.153783    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"]
Nov 03 20:20:49.172247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.167039    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-babea4-openshift-default-5d69cb64f9-cjcxz"]
Nov 03 20:20:49.267391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.267066    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:49.398407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.398263    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"]
Nov 03 20:20:49.428804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.428770    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy"
Nov 03 20:20:49.429107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429085    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy"
Nov 03 20:20:49.429316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429302    5282 cpu_manager.go:414] "containerMap: removing container" podUID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" containerName="payload-processing"
Nov 03 20:20:49.429414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429403    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" containerName="payload-processing"
Nov 03 20:20:49.429511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429501    5282 cpu_manager.go:414] "containerMap: removing container" podUID="60502a0d-1c95-4527-a002-7b71716a397b" containerName="payload-pre-processing"
Nov 03 20:20:49.429588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429577    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60502a0d-1c95-4527-a002-7b71716a397b" containerName="payload-pre-processing"
Nov 03 20:20:49.429676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429666    5282 cpu_manager.go:414] "containerMap: removing container" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy"
Nov 03 20:20:49.429749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429739    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy"
Nov 03 20:20:49.429854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429844    5282 cpu_manager.go:414] "containerMap: removing container" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy"
Nov 03 20:20:49.429967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.429954    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy"
Nov 03 20:20:49.430371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.430339    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd9726a-f255-4fe9-b09b-b1aa20027e09" containerName="payload-processing"
Nov 03 20:20:49.430505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.430493    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="60502a0d-1c95-4527-a002-7b71716a397b" containerName="payload-pre-processing"
Nov 03 20:20:49.430632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.430620    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" containerName="istio-proxy"
Nov 03 20:20:49.430745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.430734    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2ee665-2396-4632-b5eb-8ee520151e07" containerName="istio-proxy"
Nov 03 20:20:49.430866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.430855    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy"
Nov 03 20:20:49.430972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.430962    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy"
Nov 03 20:20:49.436585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.436564    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"]
Nov 03 20:20:49.436750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.436686    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.461680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.461641    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5fa49b-6181-408e-87a3-3a63ae7b4a74" path="/var/lib/kubelet/pods/9b5fa49b-6181-408e-87a3-3a63ae7b4a74/volumes"
Nov 03 20:20:49.462168 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.462151    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" path="/var/lib/kubelet/pods/a2a8e3d3-c724-43cc-8d27-c5e49d16f85e/volumes"
Nov 03 20:20:49.568254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.568179    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:20:49.569470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.569448    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 20:20:49.569526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.569496    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.592816    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.592867    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnzg\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-kube-api-access-vvnzg\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.592930    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.593012    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.593071    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.593121    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.593263    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.593400    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.594234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.593428    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e6368c-b747-4898-bed9-6d4e60dda094-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.706312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.706165    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.706660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.706634    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.706843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.706824    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e6368c-b747-4898-bed9-6d4e60dda094-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.707090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707069    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.707240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707222    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnzg\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-kube-api-access-vvnzg\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.707400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707382    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.707564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707547    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.707729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707710    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.707884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707858    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.707965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707862    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.708071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.706658    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.708452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.708428    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.708654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.707614    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e6368c-b747-4898-bed9-6d4e60dda094-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.708754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.708741    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.716070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.716042    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.717548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.717518    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.721134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.721107    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.748190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.748149    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnzg\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-kube-api-access-vvnzg\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.756872 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.756847    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:49.805225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.801573    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:49.844297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.844262    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"]
Nov 03 20:20:49.844990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.844975    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy"
Nov 03 20:20:49.845048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.844992    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a8e3d3-c724-43cc-8d27-c5e49d16f85e" containerName="istio-proxy"
Nov 03 20:20:49.851512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.850004    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:49.857804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.856386    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-456f55\""
Nov 03 20:20:49.857804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.856840    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"]
Nov 03 20:20:49.943550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.943513    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:49.956239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.956180    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"]
Nov 03 20:20:49.962225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.962017    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:49.993331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:49.987306    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-456f55-dockercfg-bn9ch\""
Nov 03 20:20:50.036553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.036515    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9fnz\" (UniqueName: \"kubernetes.io/projected/529c2f87-65fc-44d1-94dd-c9f52ff20342-kube-api-access-s9fnz\") pod \"payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:50.036760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.036576    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529c2f87-65fc-44d1-94dd-c9f52ff20342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:50.051082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.050618    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"]
Nov 03 20:20:50.141522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.139144    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fnz\" (UniqueName: \"kubernetes.io/projected/529c2f87-65fc-44d1-94dd-c9f52ff20342-kube-api-access-s9fnz\") pod \"payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:50.141522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.139213    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529c2f87-65fc-44d1-94dd-c9f52ff20342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:50.141522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.139335    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:50.141522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.139374    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpf5w\" (UniqueName: \"kubernetes.io/projected/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-kube-api-access-qpf5w\") pod \"payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:50.141522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.140659    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529c2f87-65fc-44d1-94dd-c9f52ff20342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:50.331496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.296033    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:50.331496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.296088    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpf5w\" (UniqueName: \"kubernetes.io/projected/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-kube-api-access-qpf5w\") pod \"payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:50.331496 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:20:50.296530    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e6368c_b747_4898_bed9_6d4e60dda094.slice/crio-9eb2783401b17153f0038dfabda5c1f8d97af9bcec76fd9c77bee6a9f37ef933 WatchSource:0}: Error finding container 9eb2783401b17153f0038dfabda5c1f8d97af9bcec76fd9c77bee6a9f37ef933: Status 404 returned error can't find the container with id 9eb2783401b17153f0038dfabda5c1f8d97af9bcec76fd9c77bee6a9f37ef933
Nov 03 20:20:50.331496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.297602    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:50.331496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.301623    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" event={"ID":"347cb059-324b-4249-b74b-2a18fd8d045a","Type":"ContainerStarted","Data":"f371e9464e50f7ea5a5bfc05ae38d2e0fce4c3fd5514dcdf15bc3d8c34d1a48f"}
Nov 03 20:20:50.331496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.305732    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpf5w\" (UniqueName: \"kubernetes.io/projected/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-kube-api-access-qpf5w\") pod \"payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:50.334960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.334572    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fnz\" (UniqueName: \"kubernetes.io/projected/529c2f87-65fc-44d1-94dd-c9f52ff20342-kube-api-access-s9fnz\") pod \"payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:50.340602 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.339620    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"]
Nov 03 20:20:50.384060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.382470    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:50.384060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.382565    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:50.384060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.382614    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:20:50.466259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.466144    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:50.498694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.494253    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:50.517629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.517592    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:50.570138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.569783    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 20:20:50.570138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.569830    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 20:20:50.632231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.628817    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:50.787243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.780825    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:50.824229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.808632    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:20:50.856325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.837369    5282 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:20:50.856325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.837438    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:20:50.858946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:50.858914    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:20:51.060287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.055063    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:20:51.366800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.366590    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"]
Nov 03 20:20:51.441597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.433450    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:51.574222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.573370    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 20:20:51.574222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.573428    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 20:20:51.748023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.747982    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" event={"ID":"c4e6368c-b747-4898-bed9-6d4e60dda094","Type":"ContainerStarted","Data":"a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa"}
Nov 03 20:20:51.748284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.748261    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" event={"ID":"c4e6368c-b747-4898-bed9-6d4e60dda094","Type":"ContainerStarted","Data":"9eb2783401b17153f0038dfabda5c1f8d97af9bcec76fd9c77bee6a9f37ef933"}
Nov 03 20:20:51.801237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.768914    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:51.801237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.772593    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"]
Nov 03 20:20:51.801237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.782635    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 20:20:51.801237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.782690    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 20:20:51.801237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.785598    5282 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:20:51.801237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.785645    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:20:51.845668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.845637    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:20:51.863525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:51.863463    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podStartSLOduration=2.863441597 podStartE2EDuration="2.863441597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:51.783181755 +0000 UTC m=+2444.452383247" watchObservedRunningTime="2026-11-03 20:20:51.863441597 +0000 UTC m=+2444.532643076"
Nov 03 20:20:52.196792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.196742    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:52.568324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.568229    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 20:20:52.568324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.568284    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 20:20:52.757853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.757800    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 20:20:52.758270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.757877    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 20:20:52.766166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.766135    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg" event={"ID":"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8","Type":"ContainerStarted","Data":"45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7"}
Nov 03 20:20:52.766321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.766182    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg" event={"ID":"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8","Type":"ContainerStarted","Data":"2cad1177ce450727adcef130dab98c2de93c4c904e336677bdc0aec22a3929f6"}
Nov 03 20:20:52.766321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.766231    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:20:52.768074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.768048    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg" event={"ID":"529c2f87-65fc-44d1-94dd-c9f52ff20342","Type":"ContainerStarted","Data":"292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747"}
Nov 03 20:20:52.768277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.768098    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg" event={"ID":"529c2f87-65fc-44d1-94dd-c9f52ff20342","Type":"ContainerStarted","Data":"3774d3f7bcdf28a871a1436d1136ed97a1a8995f05acd5376e8bb62eb081da6a"}
Nov 03 20:20:52.768277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.768229    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:20:52.826306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.805011    5282 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:20:52.826306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.805070    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:20:52.848925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.848856    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg" podStartSLOduration=3.8488339910000002 podStartE2EDuration="3.848833991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:52.847459591 +0000 UTC m=+2445.516661062" watchObservedRunningTime="2026-11-03 20:20:52.848833991 +0000 UTC m=+2445.518035462"
Nov 03 20:20:52.853843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.853783    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg" podStartSLOduration=3.853765563 podStartE2EDuration="3.853765563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:52.804404444 +0000 UTC m=+2445.473605916" watchObservedRunningTime="2026-11-03 20:20:52.853765563 +0000 UTC m=+2445.522967034"
Nov 03 20:20:52.886735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:52.886637    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:53.549225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.546163    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:53.570774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.570737    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 20:20:53.571098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.571049    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 20:20:53.721223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.701887    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:53.758968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.758597    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 20:20:53.758968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.758662    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 20:20:53.786664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.786625    5282 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:20:53.786963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:53.786933    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:20:54.165559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.165519    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:54.313793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.313748    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:54.568849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.568812    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 20:20:54.569035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.568866    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 20:20:54.760264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.759581    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 20:20:54.760264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.759629    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 20:20:54.808400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.808343    5282 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:20:54.808695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:54.808651    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:20:55.197431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.197295    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:55.568102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.568059    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Nov 03 20:20:55.568348 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.568110    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Nov 03 20:20:55.757671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.757635    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 20:20:55.757852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.757687    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 20:20:55.792227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.791716    5282 patch_prober.go:29] interesting pod/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Nov 03 20:20:55.792227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.791794    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Nov 03 20:20:55.950667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:55.948163    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:56.587235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.583488    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:20:56.597035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.597002    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:20:56.692225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.687706    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:56.767579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.759647    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 20:20:56.767579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.759710    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 20:20:56.906001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.905918    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:20:56.915608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.915576    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:20:56.920116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:56.920090    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:20:57.014394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:57.014147    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:57.141017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:57.140986    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:57.609713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:57.609660    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:57.757697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:57.757660    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 03 20:20:57.757859 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:57.757713    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 03 20:20:58.385580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:58.385539    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:58.801189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:58.801153    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:58.801559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:58.801534    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:58.803436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:58.803404    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:20:59.232285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:59.229376    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:20:59.899306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:20:59.899258    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:00.590508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:00.590467    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:01.217041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:01.217009    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:01.615066 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:21:01.615030    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:21:01.905069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:01.904979    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:02.039515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:02.039454    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:02.173754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:02.173659    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:02.709395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:02.709352    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:03.397036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:03.396999    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:03.814385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:03.814358    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:21:03.838078 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:03.838052    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:21:04.195165 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:04.195073    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:04.833896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:04.833857    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:05.201330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:05.201245    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:05.691047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:05.691009    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:06.479007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:06.478971    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:07.223217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:07.219554    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:07.791508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:07.791476    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:08.508150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:08.508108    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:09.193878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:09.193842    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:09.345626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:09.345587    5282 scope.go:122] "RemoveContainer" containerID="c9283a2bc3eb8cabacbe72d21ee521d2ba00c6c87ffd0c5de27a9b57c4020b3d"
Nov 03 20:21:09.800341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:09.800308    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:10.404036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:10.404001    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:10.652783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:10.652748    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:11.017156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.017124    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:11.129388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.129352    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:11.404776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.404673    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"]
Nov 03 20:21:11.413029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.412994    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.418408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.418382    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"]
Nov 03 20:21:11.421459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.421436    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a8e19e-openshift-default-dockercfg-jrlq8\""
Nov 03 20:21:11.473864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.473836    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.560277    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-podinfo\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.574305    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istiod-ca-cert\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.574470    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-token\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.574636    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-envoy\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.575061    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-credential-socket\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.575223    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-socket\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.575379    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-data\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.575445    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-certs\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.579237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.575481    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt56s\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-kube-api-access-tt56s\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682392    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-data\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682470    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-certs\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682513    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt56s\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-kube-api-access-tt56s\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682582    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-podinfo\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682631    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istiod-ca-cert\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682678    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-token\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682744    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-envoy\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682858    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-credential-socket\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.682962    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-socket\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.683577    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-socket\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.683829    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-data\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.688316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.684047    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-certs\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.690140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.690114    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istiod-ca-cert\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.717044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.697121    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-credential-socket\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.717299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.714946    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-envoy\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.726279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.726161    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-token\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.747517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.742253    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt56s\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-kube-api-access-tt56s\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.747517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.742737    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-podinfo\") pod \"e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") " pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:11.832995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.832954    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:11.987029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:11.987001    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:12.035220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:12.034526    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:12.378302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:12.378272    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"]
Nov 03 20:21:12.378446 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:21:12.378388    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ab086d_3691_4f4b_a1d2_0e7fb21c063a.slice/crio-a70d6d9c9349a11acc7a5a436ccf0d8f44be691abe3c8ba9224258573ed30200 WatchSource:0}: Error finding container a70d6d9c9349a11acc7a5a436ccf0d8f44be691abe3c8ba9224258573ed30200: Status 404 returned error can't find the container with id a70d6d9c9349a11acc7a5a436ccf0d8f44be691abe3c8ba9224258573ed30200
Nov 03 20:21:12.381326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:12.381288    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:12.381426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:12.381361    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:12.381426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:12.381393    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:12.810541 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:12.810505    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:13.150650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:13.150559    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" event={"ID":"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a","Type":"ContainerStarted","Data":"dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd"}
Nov 03 20:21:13.150650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:13.150639    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" event={"ID":"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a","Type":"ContainerStarted","Data":"a70d6d9c9349a11acc7a5a436ccf0d8f44be691abe3c8ba9224258573ed30200"}
Nov 03 20:21:13.173299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:13.173248    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podStartSLOduration=2.173234196 podStartE2EDuration="2.173234196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:13.171227789 +0000 UTC m=+2465.840429262" watchObservedRunningTime="2026-11-03 20:21:13.173234196 +0000 UTC m=+2465.842435693"
Nov 03 20:21:13.501077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:13.501041    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:14.036178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:14.036142    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:14.042189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:14.042164    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:14.094171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:14.094131    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:14.173223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:14.167509    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:14.173223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:14.170152    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:21:15.020489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:15.020458    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:15.477329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:15.477297    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:15.590484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:15.590453    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:15.816751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:15.816666    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:16.988381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:16.988324    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:17.374978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.374901    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:17.419215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.419169    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"]
Nov 03 20:21:17.424040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.424018    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:17.428383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.428352    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a8e19e\""
Nov 03 20:21:17.429061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.429043    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a8e19e-dockercfg-m8prr\""
Nov 03 20:21:17.430178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.430151    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"]
Nov 03 20:21:17.779367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.754876    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:17.792478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.792440    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:17.792680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.792506    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nggk\" (UniqueName: \"kubernetes.io/projected/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-kube-api-access-6nggk\") pod \"payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:17.894132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.894088    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:17.894375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.894156    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nggk\" (UniqueName: \"kubernetes.io/projected/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-kube-api-access-6nggk\") pod \"payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:17.894812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.894789    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:17.901701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:17.901676    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nggk\" (UniqueName: \"kubernetes.io/projected/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-kube-api-access-6nggk\") pod \"payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:18.043531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:18.043447    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:18.104120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:18.104080    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:18.331800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:18.331767    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"]
Nov 03 20:21:18.332008 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:21:18.331986    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b0b5b7_5b2a_4346_9f41_466db1e3de8f.slice/crio-2933a4fba066336d5135c6acee6447963d66d6e4734c28dca052ddb6285b346a WatchSource:0}: Error finding container 2933a4fba066336d5135c6acee6447963d66d6e4734c28dca052ddb6285b346a: Status 404 returned error can't find the container with id 2933a4fba066336d5135c6acee6447963d66d6e4734c28dca052ddb6285b346a
Nov 03 20:21:18.785489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:18.785453    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:18.911132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:18.911082    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:19.205992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.205879    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"]
Nov 03 20:21:19.211233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.211175    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.233556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.233453    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"]
Nov 03 20:21:19.259266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.257708    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww" event={"ID":"44b0b5b7-5b2a-4346-9f41-466db1e3de8f","Type":"ContainerStarted","Data":"36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4"}
Nov 03 20:21:19.259266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.257777    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww" event={"ID":"44b0b5b7-5b2a-4346-9f41-466db1e3de8f","Type":"ContainerStarted","Data":"2933a4fba066336d5135c6acee6447963d66d6e4734c28dca052ddb6285b346a"}
Nov 03 20:21:19.259266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.257804    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:19.260843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.260819    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:19.274318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.274274    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww" podStartSLOduration=2.274260544 podStartE2EDuration="2.274260544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:19.273146578 +0000 UTC m=+2471.942348049" watchObservedRunningTime="2026-11-03 20:21:19.274260544 +0000 UTC m=+2471.943462013"
Nov 03 20:21:19.330230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330168    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.330386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330320    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.330386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330354    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.330487 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330465    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.330523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330501    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.330655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330638    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.330793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330772    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2hfl\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-kube-api-access-s2hfl\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.330908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330893    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.331002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.330988    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.439271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.439227    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.439665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.439642    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2hfl\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-kube-api-access-s2hfl\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.439883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.439868    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.440408    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.442075    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.442800    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.443022    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.447554    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.447616    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.447654    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.447725    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.447749    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.448923    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.449908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.449167    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.452534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.452508    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.453008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.452975    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2hfl\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-kube-api-access-s2hfl\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.456585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.456282    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.459063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.459042    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.538536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.538502    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:19.785218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.776948    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:19.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.954610    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:19.965302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.961214    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"]
Nov 03 20:21:19.999542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.983853    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:19.999542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.983944    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:19.999542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:19.983989    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:20.268969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:20.268929    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" event={"ID":"690ea14a-8ce1-4776-a478-fd9a2b66b4c9","Type":"ContainerStarted","Data":"57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b"}
Nov 03 20:21:20.269507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:20.269013    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" event={"ID":"690ea14a-8ce1-4776-a478-fd9a2b66b4c9","Type":"ContainerStarted","Data":"abc3a986561bd73cde0a7c2acea76d188f683fbdb926b379ec9f21e431d1c91a"}
Nov 03 20:21:20.293385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:20.293325    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podStartSLOduration=1.293310802 podStartE2EDuration="1.293310802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:20.289576823 +0000 UTC m=+2472.958778292" watchObservedRunningTime="2026-11-03 20:21:20.293310802 +0000 UTC m=+2472.962512296"
Nov 03 20:21:20.538818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:20.538777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:20.831919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:20.831824    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:21.022247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:21.022192    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:21.154029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:21.153941    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:21.540877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:21.540840    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:21:21.541300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:21.540896    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 20:21:21.637737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:21.637703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:21.838791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:21.838712    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:22.538905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:22.538864    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:21:22.539087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:22.538943    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 20:21:22.851874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:22.851784    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:23.402295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:23.402249    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:23.530108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:23.530071    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:23.539250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:23.539222    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:21:23.539360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:23.539273    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 20:21:23.909232 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:23.909163    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:24.539239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:24.539191    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:21:24.539393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:24.539255    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 20:21:24.810187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:24.810110    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:24.966112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:24.966078    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"]
Nov 03 20:21:24.967170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:24.967138    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:24.988417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:24.988387    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:25.036225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.002667    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b55105\""
Nov 03 20:21:25.036225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.005447    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b55105-dockercfg-9m8kd\""
Nov 03 20:21:25.036225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.007133    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"]
Nov 03 20:21:25.083743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.083712    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:25.343185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.343064    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxqv\" (UniqueName: \"kubernetes.io/projected/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-kube-api-access-rdxqv\") pod \"payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:25.344624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.344596    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:25.379445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.377545    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:21:25.461529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.461485    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxqv\" (UniqueName: \"kubernetes.io/projected/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-kube-api-access-rdxqv\") pod \"payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:25.461771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.461753    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:25.487266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.463340    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:25.540645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.540605    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:21:25.540916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.540884    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 20:21:25.556822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.556789    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxqv\" (UniqueName: \"kubernetes.io/projected/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-kube-api-access-rdxqv\") pod \"payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:25.686891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:25.686558    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:26.068904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.068870    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"]
Nov 03 20:21:26.102003 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:21:26.101966    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7c63cf_607d_467c_9239_9ed4e1e3a6ba.slice/crio-3ac3b046e88b1fb9092b497aefecfd9226915bf8d44706cdd620b73923e816f5 WatchSource:0}: Error finding container 3ac3b046e88b1fb9092b497aefecfd9226915bf8d44706cdd620b73923e816f5: Status 404 returned error can't find the container with id 3ac3b046e88b1fb9092b497aefecfd9226915bf8d44706cdd620b73923e816f5
Nov 03 20:21:26.330272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.325589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:26.444222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.433882    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss" event={"ID":"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba","Type":"ContainerStarted","Data":"6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f"}
Nov 03 20:21:26.444222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.433947    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss" event={"ID":"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba","Type":"ContainerStarted","Data":"3ac3b046e88b1fb9092b497aefecfd9226915bf8d44706cdd620b73923e816f5"}
Nov 03 20:21:26.444222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.435418    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:26.494219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.484547    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss" podStartSLOduration=2.484525966 podStartE2EDuration="2.484525966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:26.48270423 +0000 UTC m=+2479.151905702" watchObservedRunningTime="2026-11-03 20:21:26.484525966 +0000 UTC m=+2479.153727432"
Nov 03 20:21:26.555227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.542644    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:21:26.555227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:26.542713    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 20:21:27.322361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:27.322311    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:27.554241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:27.548374    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:21:27.554241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:27.548433    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 20:21:28.109766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:28.109730    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:28.543857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:28.543827    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:28.544349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:28.544328    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:28.545150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:28.545131    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:21:29.458088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:29.458043    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:30.272626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:30.272592    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:21:30.493774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:30.493661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:31.331474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:31.331441    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:31.627384 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:21:31.627298    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:21:32.131603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:32.131572    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:33.072269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:33.072232    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:34.035876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.035838    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:34.229287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.229250    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"]
Nov 03 20:21:34.235844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.235816    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.253668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.253623    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.253858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.253685    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.253858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.253734    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18f42331-cba4-451d-b86c-89dc9e983332-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.253982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.253919    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.254079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.254058    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbwf\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-kube-api-access-thbwf\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.254149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.254092    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.254149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.254124    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.254313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.254151    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18f42331-cba4-451d-b86c-89dc9e983332-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.254373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.254344    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.268792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.268740    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"]
Nov 03 20:21:34.356278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356153    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356332    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbwf\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-kube-api-access-thbwf\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356362    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356395    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356419    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18f42331-cba4-451d-b86c-89dc9e983332-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356543    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356601    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-data\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356697    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356734    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.356801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.356778    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18f42331-cba4-451d-b86c-89dc9e983332-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.357307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.357100    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-certs\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.357307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.357169    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-credential-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.357307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.357255    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-socket\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.357682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.357649    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18f42331-cba4-451d-b86c-89dc9e983332-istiod-ca-cert\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.359264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.359238    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-envoy\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.359451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.359434    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18f42331-cba4-451d-b86c-89dc9e983332-istio-podinfo\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.367138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.367112    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbwf\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-kube-api-access-thbwf\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.367320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.367229    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-istio-token\") pod \"e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") " pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.568711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.568676    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:34.848222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.841921    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:34.899248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.897669    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"]
Nov 03 20:21:34.903446 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:21:34.903408    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f42331_cba4_451d_b86c_89dc9e983332.slice/crio-b2fd137bd6728a9e647f4cb863a026da8cf0b982f356b9bb999470a838ed5ac0 WatchSource:0}: Error finding container b2fd137bd6728a9e647f4cb863a026da8cf0b982f356b9bb999470a838ed5ac0: Status 404 returned error can't find the container with id b2fd137bd6728a9e647f4cb863a026da8cf0b982f356b9bb999470a838ed5ac0
Nov 03 20:21:34.907772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.907729    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:34.907890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.907805    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:34.907890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.907844    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:34.959477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:34.959450    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:35.138436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:35.138394    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:21:35.138648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:35.138464    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:35.662380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:35.662292    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" event={"ID":"18f42331-cba4-451d-b86c-89dc9e983332","Type":"ContainerStarted","Data":"f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac"}
Nov 03 20:21:35.662380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:35.662358    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" event={"ID":"18f42331-cba4-451d-b86c-89dc9e983332","Type":"ContainerStarted","Data":"b2fd137bd6728a9e647f4cb863a026da8cf0b982f356b9bb999470a838ed5ac0"}
Nov 03 20:21:35.688257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:35.688138    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podStartSLOduration=1.688117294 podStartE2EDuration="1.688117294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:35.68205755 +0000 UTC m=+2488.351259021" watchObservedRunningTime="2026-11-03 20:21:35.688117294 +0000 UTC m=+2488.357318766"
Nov 03 20:21:35.925175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:35.925089    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:36.570056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:36.570018    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:36.571456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:36.571432    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:21:36.571566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:36.571481    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:21:36.966382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:36.966275    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:37.569511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:37.569475    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:21:37.569705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:37.569528    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:21:37.902581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:37.902438    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:38.527823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.527795    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:21:38.569285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.569247    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:21:38.569450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.569306    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:21:38.710241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.710163    5282 generic.go:356] "Generic (PLEG): container finished" podID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerID="046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6" exitCode=0
Nov 03 20:21:38.710446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.710271    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" event={"ID":"4f9f6315-7999-43a5-8ca3-6434b1058d58","Type":"ContainerDied","Data":"046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6"}
Nov 03 20:21:38.710816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.710793    5282 scope.go:122] "RemoveContainer" containerID="046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6"
Nov 03 20:21:38.712757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.712713    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:38.712866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.712785    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:38.712866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.712819    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:38.791285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:38.780662    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:39.261406 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.261372    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"]
Nov 03 20:21:39.266143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.266117    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.277491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.277467    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a476f8-openshift-default-dockercfg-lsx9m\""
Nov 03 20:21:39.306719 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.306689    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"]
Nov 03 20:21:39.315523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315478    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.315702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315528    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsfb\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-kube-api-access-nxsfb\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.315702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315558    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.315702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315616    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.315702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315691    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.315940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315715    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.315940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315764    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.315940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.315860    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.316107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.316004    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.331040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.330997    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"]
Nov 03 20:21:39.331222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.331129    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.331453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.331420    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"]
Nov 03 20:21:39.417423 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417388    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.417594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417497    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2w5\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-kube-api-access-4c2w5\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.417594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417528    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.417594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417549    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.417594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417575    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsfb\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-kube-api-access-nxsfb\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.417740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417645    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.417740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417727    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.417807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417765    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.417843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417803    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.417843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417832    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.417909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417887    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebd8ea9-701c-4476-9380-9856f565e819-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.418005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417979    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.417987    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418098    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418165    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418261    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.418458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418292    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418309    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebd8ea9-701c-4476-9380-9856f565e819-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.418458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418443    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418469    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418620    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.418831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.418776    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.419364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.419341    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.420175 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.420152    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.422079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.422052    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.425523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.425499    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.425749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.425726    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsfb\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-kube-api-access-nxsfb\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.520643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.520542    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.520643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.520603    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebd8ea9-701c-4476-9380-9856f565e819-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.520862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.520830    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.520910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.520899    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2w5\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-kube-api-access-4c2w5\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.520952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.520935    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521008    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521053    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521081    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521130    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebd8ea9-701c-4476-9380-9856f565e819-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521012    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521183    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521409    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebd8ea9-701c-4476-9380-9856f565e819-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521549    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.521670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.521645    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.523386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.523367    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.523626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.523606    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebd8ea9-701c-4476-9380-9856f565e819-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.529350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.529326    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2w5\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-kube-api-access-4c2w5\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.529646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.529347    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.569509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.569475    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:21:39.569666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.569525    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:21:39.632874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.632837    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:39.649270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.649243    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:39.732054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.730462    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" event={"ID":"4f9f6315-7999-43a5-8ca3-6434b1058d58","Type":"ContainerStarted","Data":"a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1"}
Nov 03 20:21:39.744809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:39.742105    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:40.156085 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:21:40.156052    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0213ef7_a48f_4746_8c08_57b1a3961c1d.slice/crio-7400f9866268d6c9cbdced5c65fd43ade15b59976802bc05a272d2275f8baf73 WatchSource:0}: Error finding container 7400f9866268d6c9cbdced5c65fd43ade15b59976802bc05a272d2275f8baf73: Status 404 returned error can't find the container with id 7400f9866268d6c9cbdced5c65fd43ade15b59976802bc05a272d2275f8baf73
Nov 03 20:21:40.156085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.156057    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"]
Nov 03 20:21:40.158932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.158898    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:40.159033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.158982    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:40.159033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.159019    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:40.241115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.241073    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:40.241278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.241150    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:40.241278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.241183    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:40.241412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.241386    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"]
Nov 03 20:21:40.267015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.266988    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:21:40.267159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.267139    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:21:40.267246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.267184    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 20:21:40.569183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.569148    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:21:40.569783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.569753    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:21:40.748654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.748542    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" event={"ID":"8ebd8ea9-701c-4476-9380-9856f565e819","Type":"ContainerStarted","Data":"338ee561b709de838fc6510528e23d2198b80d5ce138a8276c9e64d886505ef3"}
Nov 03 20:21:40.748654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.748614    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" event={"ID":"8ebd8ea9-701c-4476-9380-9856f565e819","Type":"ContainerStarted","Data":"7cad4af3322bfb23b0105bdb1f70ca8c8aac14cb0e680cb2c049df65accaabd9"}
Nov 03 20:21:40.750844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.750813    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" event={"ID":"f0213ef7-a48f-4746-8c08-57b1a3961c1d","Type":"ContainerStarted","Data":"a04605cd504daad2a41c8a0dcc6b62e0a94f3e39507f1769aa89bb2966d19665"}
Nov 03 20:21:40.750968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.750869    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" event={"ID":"f0213ef7-a48f-4746-8c08-57b1a3961c1d","Type":"ContainerStarted","Data":"7400f9866268d6c9cbdced5c65fd43ade15b59976802bc05a272d2275f8baf73"}
Nov 03 20:21:40.768660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.768614    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podStartSLOduration=1.768603067 podStartE2EDuration="1.768603067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:40.766041 +0000 UTC m=+2493.435242497" watchObservedRunningTime="2026-11-03 20:21:40.768603067 +0000 UTC m=+2493.437804531"
Nov 03 20:21:40.813604 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.812830    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podStartSLOduration=1.812809033 podStartE2EDuration="1.812809033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:40.791547996 +0000 UTC m=+2493.460749467" watchObservedRunningTime="2026-11-03 20:21:40.812809033 +0000 UTC m=+2493.482010502"
Nov 03 20:21:40.817227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:40.814074    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:41.266493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.266454    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:21:41.266709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.266510    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 20:21:41.578454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.578319    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:21:41.579114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.579079    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 03 20:21:41.642605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.642577    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:41.643790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.643763    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:21:41.643988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.643966    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 20:21:41.651429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.651401    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:41.653285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.653256    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:21:41.653867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.653836    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 20:21:41.735666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:41.735633    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:42.310245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.301540    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:21:42.310245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.302010    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 20:21:42.668297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.667026    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:21:42.682355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.682303    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 20:21:42.706225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.694583    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:21:42.706225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.694667    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 20:21:42.766263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.746299    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:42.766263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.749338    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:42.766263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.759660    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:21:42.943283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:42.845665    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:43.045989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.043366    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 20:21:43.298274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.273937    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:43.335271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.298544    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:21:43.360370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.360324    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 20:21:43.364674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.358406    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:43.494226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.493591    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:43.526876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.522441    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:43.665521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.665411    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:21:43.674218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.665761    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 20:21:43.684710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.678429    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:21:43.689750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.689316    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 20:21:43.747267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:43.739981    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:44.064722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.064674    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" podUID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.272229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.269754    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:21:44.272229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.269819    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 20:21:44.318799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.318691    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Nov 03 20:21:44.361947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.361913    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:44.387219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.385832    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:44.387219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.386630    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:21:44.642956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.642860    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:21:44.643252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.643223    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 20:21:44.680561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.680524    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:21:44.719319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.719272    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 20:21:44.820326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:44.820285    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:21:45.219040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.218852    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:45.279525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.279487    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:21:45.279886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.279856    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 20:21:45.443850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.443807    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:45.635223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.634375    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:21:45.635223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.634440    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 20:21:45.650369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.650333    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:21:45.650811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.650781    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 20:21:45.956635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:45.956544    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:46.266753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:46.266718    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Nov 03 20:21:46.267037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:46.266998    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Nov 03 20:21:46.634328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:46.634229    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:21:46.634583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:46.634553    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 20:21:46.650098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:46.650061    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:21:46.650306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:46.650285    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 20:21:46.696229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:46.695615    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:47.272889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.272855    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:21:47.273368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.273160    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:21:47.320533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.320499    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:21:47.435359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.435152    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:47.638432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.638338    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:21:47.638697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.638670    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 20:21:47.650192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.650157    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:21:47.650352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:47.650221    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 20:21:48.014675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.014639    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:48.705030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.704997    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:48.788271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.782945    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:48.788891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.788865    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:48.789423 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.789401    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:48.791114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.791094    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:48.792450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.792426    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:21:48.803331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:48.803302    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:21:49.588758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:49.588725    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:50.281389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:50.281351    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:50.958150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:50.958119    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:51.704735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:51.704703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:52.382425 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:52.382392    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:53.534935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:53.534893    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:54.299670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.299634    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"]
Nov 03 20:21:54.305759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.305736    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.318112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.318082    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"]
Nov 03 20:21:54.397274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397235    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397300    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397351    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08629be-9d3a-4e49-8935-e9f089de7ce9-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397424    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397454    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397655    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397699    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397805    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.397881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.397866    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxj4\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-kube-api-access-bfxj4\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499528    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499574    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499590    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08629be-9d3a-4e49-8935-e9f089de7ce9-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499623    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499646    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499804    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499837    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.499971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499903    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.500131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.499969    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxj4\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-kube-api-access-bfxj4\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.500189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.500149    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-credential-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.500331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.500310    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-certs\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.500454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.500423    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-socket\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.500568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.500430    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-data\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.500568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.500550    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08629be-9d3a-4e49-8935-e9f089de7ce9-istiod-ca-cert\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.502129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.502106    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-podinfo\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.502312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.502293    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-envoy\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.510849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.510828    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-token\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.511040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.511017    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxj4\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-kube-api-access-bfxj4\") pod \"e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") " pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:54.590619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.590526    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:54.632034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:54.631305    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:55.006281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.006249    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"]
Nov 03 20:21:55.006399 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:21:55.006365    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08629be_9d3a_4e49_8935_e9f089de7ce9.slice/crio-81672d88762961aa0a254302f1145ebb0eef9d6c9ac316d145b7b8f6e43fb120 WatchSource:0}: Error finding container 81672d88762961aa0a254302f1145ebb0eef9d6c9ac316d145b7b8f6e43fb120: Status 404 returned error can't find the container with id 81672d88762961aa0a254302f1145ebb0eef9d6c9ac316d145b7b8f6e43fb120
Nov 03 20:21:55.009382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.009348    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:55.009498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.009427    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:55.009498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.009475    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:21:55.325148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.325067    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:55.436030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.435991    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" event={"ID":"c08629be-9d3a-4e49-8935-e9f089de7ce9","Type":"ContainerStarted","Data":"df2198c0e2922a350431ae8b48ea1150fb619f8b8d2e5f1281070ed0ca95e4db"}
Nov 03 20:21:55.436223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.436053    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" event={"ID":"c08629be-9d3a-4e49-8935-e9f089de7ce9","Type":"ContainerStarted","Data":"81672d88762961aa0a254302f1145ebb0eef9d6c9ac316d145b7b8f6e43fb120"}
Nov 03 20:21:55.460028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.459952    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podStartSLOduration=1.4599317410000001 podStartE2EDuration="1.459931741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:55.455449727 +0000 UTC m=+2508.124651198" watchObservedRunningTime="2026-11-03 20:21:55.459931741 +0000 UTC m=+2508.129133212"
Nov 03 20:21:55.632383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.632291    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:21:55.871015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:55.870977    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:56.370553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:56.370521    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:56.633952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:56.633875    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:21:56.633952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:56.633935    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 20:21:57.421156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:57.421122    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:57.632790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:57.632755    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:21:57.632940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:57.632818    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 20:21:58.427419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.427381    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:58.632681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.632642    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:21:58.632887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.632699    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 20:21:58.827046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.827008    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"]
Nov 03 20:21:58.838222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.834644    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:58.844876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.844100    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ff9f5\"/\"openshift-service-ca.crt\""
Nov 03 20:21:58.844876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.844240    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ff9f5\"/\"kube-root-ca.crt\""
Nov 03 20:21:58.845451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.845429    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ff9f5\"/\"default-dockercfg-2rdwf\""
Nov 03 20:21:58.845661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.845642    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ff9f5\"/\"auth-test-model-9ff9f5-kserve-self-signed-certs\""
Nov 03 20:21:58.848134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.848113    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"]
Nov 03 20:21:58.937442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.937407    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:58.945343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.945317    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-tmp-dir\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:58.945485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.945464    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-dshm\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:58.945601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.945586    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kn2\" (UniqueName: \"kubernetes.io/projected/2d928738-b657-4333-b311-13232a0a4aa5-kube-api-access-s6kn2\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:58.945638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.945628    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d928738-b657-4333-b311-13232a0a4aa5-tls-certs\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:58.945818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.945798    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-model-cache\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:58.945859 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:58.945839    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-home\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.051222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.050364    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-dshm\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.051222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.050540    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kn2\" (UniqueName: \"kubernetes.io/projected/2d928738-b657-4333-b311-13232a0a4aa5-kube-api-access-s6kn2\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.051222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.050600    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d928738-b657-4333-b311-13232a0a4aa5-tls-certs\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.062405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.059368    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-model-cache\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.062405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.059443    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-home\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.062405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.059492    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-tmp-dir\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.062405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.059958    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-tmp-dir\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.062405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.060259    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-model-cache\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.062405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.060559    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-home\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.113516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.112362    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kn2\" (UniqueName: \"kubernetes.io/projected/2d928738-b657-4333-b311-13232a0a4aa5-kube-api-access-s6kn2\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.134216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.127025    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d928738-b657-4333-b311-13232a0a4aa5-tls-certs\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.134216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.127436    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-dshm\") pod \"auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") " pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.172229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.168954    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:21:59.616944 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:21:59.616916    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d928738_b657_4333_b311_13232a0a4aa5.slice/crio-31d631947274b499021beb1b2f71424bccfa0750b993f93afc12132632eae0b5 WatchSource:0}: Error finding container 31d631947274b499021beb1b2f71424bccfa0750b993f93afc12132632eae0b5: Status 404 returned error can't find the container with id 31d631947274b499021beb1b2f71424bccfa0750b993f93afc12132632eae0b5
Nov 03 20:21:59.617412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.617243    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"]
Nov 03 20:21:59.632376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.632344    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:21:59.632509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.632427    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 20:21:59.636531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.636506    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"]
Nov 03 20:21:59.663798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.659341    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.699224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.698590    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-0ffd89-openshift-default-dockercfg-rq6r9\""
Nov 03 20:21:59.705291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.705249    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"]
Nov 03 20:21:59.718593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.718173    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719321    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-token\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719388    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istiod-ca-cert\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719484    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-certs\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719554    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-credential-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719584    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-envoy\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719619    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-podinfo\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719660    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvr5\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-kube-api-access-swvr5\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719720    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.725143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.719867    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-data\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.909735    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-data\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.909854    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-token\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.909901    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istiod-ca-cert\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.909985    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-certs\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910054    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-credential-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910084    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-envoy\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910120    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-podinfo\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910157    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-data\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910165    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvr5\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-kube-api-access-swvr5\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.910603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910294    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.911283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910724    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.911283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.910955    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-certs\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.911449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.911425    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istiod-ca-cert\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.911737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.911715    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-credential-socket\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.913047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.913023    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-envoy\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.913957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.913937    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-podinfo\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.920184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.920156    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-token\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:21:59.920921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:21:59.920904    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvr5\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-kube-api-access-swvr5\") pod \"e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") " pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:22:00.122322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.122281    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:22:00.434261 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:22:00.434230    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48c0353_24d5_43a4_ba99_ee8d666ad6a2.slice/crio-f9186da31ae65b5f35bc09d0931ee35d199264820389c6a4a2182104174d0107 WatchSource:0}: Error finding container f9186da31ae65b5f35bc09d0931ee35d199264820389c6a4a2182104174d0107: Status 404 returned error can't find the container with id f9186da31ae65b5f35bc09d0931ee35d199264820389c6a4a2182104174d0107
Nov 03 20:22:00.434773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.434747    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"]
Nov 03 20:22:00.437345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.437312    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:22:00.437439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.437392    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:22:00.437510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.437436    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:22:00.623446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.623410    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" event={"ID":"f48c0353-24d5-43a4-ba99-ee8d666ad6a2","Type":"ContainerStarted","Data":"7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c"}
Nov 03 20:22:00.623962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.623466    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" event={"ID":"f48c0353-24d5-43a4-ba99-ee8d666ad6a2","Type":"ContainerStarted","Data":"f9186da31ae65b5f35bc09d0931ee35d199264820389c6a4a2182104174d0107"}
Nov 03 20:22:00.626268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.626242    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5" event={"ID":"2d928738-b657-4333-b311-13232a0a4aa5","Type":"ContainerStarted","Data":"94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff"}
Nov 03 20:22:00.626384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.626277    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5" event={"ID":"2d928738-b657-4333-b311-13232a0a4aa5","Type":"ContainerStarted","Data":"31d631947274b499021beb1b2f71424bccfa0750b993f93afc12132632eae0b5"}
Nov 03 20:22:00.632512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.632482    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:22:00.632613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.632533    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 20:22:00.648472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.648428    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podStartSLOduration=1.648414452 podStartE2EDuration="1.648414452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:00.64339358 +0000 UTC m=+2513.312595075" watchObservedRunningTime="2026-11-03 20:22:00.648414452 +0000 UTC m=+2513.317615921"
Nov 03 20:22:00.662907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.662853    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5" podStartSLOduration=2.662841145 podStartE2EDuration="2.662841145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:00.659417529 +0000 UTC m=+2513.328618999" watchObservedRunningTime="2026-11-03 20:22:00.662841145 +0000 UTC m=+2513.332042614"
Nov 03 20:22:00.802252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:00.799178    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:01.123309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.123216    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:22:01.124527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.124501    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:22:01.124648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.124549    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 20:22:01.169831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.169795    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:22:01.618294 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:22:01.618255    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:22:01.701324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.700210    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:22:01.701324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.701186    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:22:01.702611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.702592    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:22:01.729934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:01.729894    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:02.127112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:02.126956    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:22:02.127112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:02.127027    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 20:22:03.122800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:03.122758    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:22:03.123315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:03.122817    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 20:22:03.193813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:03.193661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:04.042821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:04.042785    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:04.130226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:04.129305    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:22:04.130226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:04.129370    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 20:22:04.237804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:04.237770    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:04.923157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:04.923102    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:05.123345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:05.123301    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:22:05.123555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:05.123366    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 20:22:05.730264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:05.730222    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:06.123722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:06.123624    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:22:06.123722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:06.123683    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 20:22:06.402749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:06.402663    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:07.111660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:07.111625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:07.122989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:07.122963    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:22:07.123129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:07.123022    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 20:22:07.787451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:07.787414    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:08.130728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:08.130643    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:22:08.131684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:08.131661    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:22:08.133012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:08.132993    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:22:08.518019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:08.517989    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:09.169712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:09.169674    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:22:09.182573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:09.182541    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:22:09.302342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:09.302306    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:09.784310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:09.784279    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:22:10.090597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:10.090504    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:10.792629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:10.792595    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:11.013963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.013920    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"]
Nov 03 20:22:11.021366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.021333    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.026065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.026035    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-456f55\"/\"default-dockercfg-hzwlc\""
Nov 03 20:22:11.026641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.026619    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-456f55\"/\"openshift-service-ca.crt\""
Nov 03 20:22:11.026797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.026668    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-456f55\"/\"auth-test-model-456f55-kserve-self-signed-certs\""
Nov 03 20:22:11.026871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.026687    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-456f55\"/\"kube-root-ca.crt\""
Nov 03 20:22:11.029056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.029030    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"]
Nov 03 20:22:11.084334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.084252    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:11.099023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.098993    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-home\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.099158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.099041    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7hn\" (UniqueName: \"kubernetes.io/projected/76eaf966-caf0-4394-9f02-488f93755253-kube-api-access-mq7hn\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.099158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.099137    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76eaf966-caf0-4394-9f02-488f93755253-tls-certs\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.099292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.099171    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-model-cache\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.099456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.099438    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-dshm\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.099568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.099554    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-tmp-dir\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.209347    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-tmp-dir\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.209542    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-home\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.209585    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7hn\" (UniqueName: \"kubernetes.io/projected/76eaf966-caf0-4394-9f02-488f93755253-kube-api-access-mq7hn\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.209624    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76eaf966-caf0-4394-9f02-488f93755253-tls-certs\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.209658    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-model-cache\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.209871    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-dshm\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.211120    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-tmp-dir\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.212225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.211494    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-home\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.221280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.221248    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-model-cache\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.226297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.225317    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-dshm\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.240221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.235103    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7hn\" (UniqueName: \"kubernetes.io/projected/76eaf966-caf0-4394-9f02-488f93755253-kube-api-access-mq7hn\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.244033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.244006    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76eaf966-caf0-4394-9f02-488f93755253-tls-certs\") pod \"auth-test-model-456f55-kserve-68dbcfc989-hr96z\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") " pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.339638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.339548    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:11.620071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.619987    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:11.730540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.727951    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"]
Nov 03 20:22:11.791704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:11.791664    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z" event={"ID":"76eaf966-caf0-4394-9f02-488f93755253","Type":"ContainerStarted","Data":"36a399446b4bb7df90f409ac4b545fa87590ad7a5f32dc05ce169851cf96f825"}
Nov 03 20:22:12.309450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:12.309413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:12.800975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:12.800936    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z" event={"ID":"76eaf966-caf0-4394-9f02-488f93755253","Type":"ContainerStarted","Data":"a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb"}
Nov 03 20:22:12.835967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:12.835907    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z" podStartSLOduration=2.835889581 podStartE2EDuration="2.835889581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:12.815748055 +0000 UTC m=+2525.484949517" watchObservedRunningTime="2026-11-03 20:22:12.835889581 +0000 UTC m=+2525.505091050"
Nov 03 20:22:12.996468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:12.996432    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:13.344218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:13.341300    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:14.003118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:14.003046    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:15.211416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:15.211386    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:16.085698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:16.085608    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:16.911750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:16.910471    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:17.509411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:17.509379    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:18.234082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:18.234046    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:18.924700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:18.924643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:19.068940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:19.068746    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:19.710422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:19.710383    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:20.711386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:20.711332    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:21.340539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:21.340496    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:21.353934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:21.353904    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:21.487410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:21.487371    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:21.724040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:21.723943    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:21.970099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:21.970067    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:22:22.290584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:22.290547    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:23.173170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:23.173133    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:23.956036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:23.955973    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:24.816222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:24.808886    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:25.487988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:25.487954    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:26.242988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:26.242949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:27.104256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:27.104182    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:28.003160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:28.003125    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:28.737319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:28.734627    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:29.421554 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:29.421519    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:30.296912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:30.296879    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:31.024814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:31.024783    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:31.616364 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:22:31.616325    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:22:31.792553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:31.792513    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:32.495738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:32.495705    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:32.628398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:32.628362    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:33.283563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:33.283530    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:34.095661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:34.095629    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:34.916042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:34.916005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:35.554828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:35.554798    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:35.680888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:35.680853    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:36.252557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:36.252522    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:37.015990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:37.015950    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:37.678635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:37.678602    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:22:37.907358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:37.907322    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:38.791314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:38.791274    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:39.905664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:39.905625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:40.698268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:40.695591    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:41.572220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:41.572161    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:42.111166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:42.111133    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"]
Nov 03 20:22:42.165984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:42.165950    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:42.518802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:42.518760    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:43.198684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:43.198648    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:43.906234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:43.889401    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:44.712496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:44.712466    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:45.458092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:45.458042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:46.049792 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:46.049752    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:46.667404 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:46.667363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:47.263551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:47.263509    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:47.663728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:47.663644    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:48.737221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:48.734225    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:49.392062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:49.392032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:50.122058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.121974    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:50.516910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.516875    5282 generic.go:356] "Generic (PLEG): container finished" podID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerID="dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd" exitCode=0
Nov 03 20:22:50.517082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.516918    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" event={"ID":"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a","Type":"ContainerDied","Data":"dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd"}
Nov 03 20:22:50.517354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.517338    5282 scope.go:122] "RemoveContainer" containerID="dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd"
Nov 03 20:22:50.518956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.518925    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:22:50.519040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.518999    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:22:50.519040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.519033    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:22:50.884840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:50.884754    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:51.526575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:51.526539    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" event={"ID":"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a","Type":"ContainerStarted","Data":"83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b"}
Nov 03 20:22:51.701096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:51.701058    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:52.035856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:52.035819    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:22:52.036049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:52.035951    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:22:52.036049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:52.036004    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 20:22:52.397882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:52.397792    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:53.026888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:53.026859    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:53.036144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:53.036114    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:22:53.036530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:53.036455    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 20:22:53.712445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:53.712410    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:54.036297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:54.036259    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:22:54.036735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:54.036324    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 20:22:54.503247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:54.503191    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:55.035751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:55.035718    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:22:55.035933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:55.035794    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 20:22:55.199773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:55.199733    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:55.924343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:55.924299    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:56.040753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:56.040712    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:22:56.041093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:56.041044    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 20:22:56.695811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:56.695769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:57.036345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.036313    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:22:57.036529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.036378    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 03 20:22:57.155673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.155636    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5" podUID="2d928738-b657-4333-b311-13232a0a4aa5" containerName="main" containerID="cri-o://94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff" gracePeriod=45
Nov 03 20:22:57.438252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.438141    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:57.563146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.563115    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:22:57.623219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.623108    5282 generic.go:356] "Generic (PLEG): container finished" podID="2d928738-b657-4333-b311-13232a0a4aa5" containerID="94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff" exitCode=0
Nov 03 20:22:57.623219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.623152    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"
Nov 03 20:22:57.623219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.623150    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5" event={"ID":"2d928738-b657-4333-b311-13232a0a4aa5","Type":"ContainerDied","Data":"94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff"}
Nov 03 20:22:57.623498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.623220    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5" event={"ID":"2d928738-b657-4333-b311-13232a0a4aa5","Type":"ContainerDied","Data":"31d631947274b499021beb1b2f71424bccfa0750b993f93afc12132632eae0b5"}
Nov 03 20:22:57.623498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.623244    5282 scope.go:122] "RemoveContainer" containerID="94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff"
Nov 03 20:22:57.638442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.638416    5282 scope.go:122] "RemoveContainer" containerID="94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff"
Nov 03 20:22:57.638824 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:22:57.638796    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff\": container with ID starting with 94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff not found: ID does not exist" containerID="94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff"
Nov 03 20:22:57.638941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.638849    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff"} err="failed to get container status \"94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff\": rpc error: code = NotFound desc = could not find container \"94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff\": container with ID starting with 94731efd1ed81de4795515d28f1b6e8db17112133a0d2b7c21752354a0a160ff not found: ID does not exist"
Nov 03 20:22:57.642559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.642529    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d928738-b657-4333-b311-13232a0a4aa5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d928738-b657-4333-b311-13232a0a4aa5-tls-certs\") pod \"2d928738-b657-4333-b311-13232a0a4aa5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") "
Nov 03 20:22:57.642695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.642589    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-home\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-home\") pod \"2d928738-b657-4333-b311-13232a0a4aa5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") "
Nov 03 20:22:57.642695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.642684    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-tmp-dir\") pod \"2d928738-b657-4333-b311-13232a0a4aa5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") "
Nov 03 20:22:57.643012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.642985    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-tmp-dir" pod "2d928738-b657-4333-b311-13232a0a4aa5" (UID: "2d928738-b657-4333-b311-13232a0a4aa5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:57.643352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.643330    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-home" pod "2d928738-b657-4333-b311-13232a0a4aa5" (UID: "2d928738-b657-4333-b311-13232a0a4aa5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:57.643503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.643476    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-dshm\") pod \"2d928738-b657-4333-b311-13232a0a4aa5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") "
Nov 03 20:22:57.643730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.643707    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d928738-b657-4333-b311-13232a0a4aa5-kube-api-access-s6kn2\" (UniqueName: \"kubernetes.io/projected/2d928738-b657-4333-b311-13232a0a4aa5-kube-api-access-s6kn2\") pod \"2d928738-b657-4333-b311-13232a0a4aa5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") "
Nov 03 20:22:57.643794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.643759    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-model-cache\") pod \"2d928738-b657-4333-b311-13232a0a4aa5\" (UID: \"2d928738-b657-4333-b311-13232a0a4aa5\") "
Nov 03 20:22:57.644646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.644621    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:57.644733 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.644652    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:57.644804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.644742    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-model-cache" pod "2d928738-b657-4333-b311-13232a0a4aa5" (UID: "2d928738-b657-4333-b311-13232a0a4aa5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:57.646379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.646357    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d928738-b657-4333-b311-13232a0a4aa5-tls-certs" pod "2d928738-b657-4333-b311-13232a0a4aa5" (UID: "2d928738-b657-4333-b311-13232a0a4aa5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:57.646658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.646622    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-dshm" pod "2d928738-b657-4333-b311-13232a0a4aa5" (UID: "2d928738-b657-4333-b311-13232a0a4aa5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:57.648631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.648601    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d928738-b657-4333-b311-13232a0a4aa5-kube-api-access-s6kn2" pod "2d928738-b657-4333-b311-13232a0a4aa5" (UID: "2d928738-b657-4333-b311-13232a0a4aa5"). InnerVolumeSpecName "kube-api-access-s6kn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:57.664025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.664002    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"]
Nov 03 20:22:57.682172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.682137    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"]
Nov 03 20:22:57.746944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.746906    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6kn2\" (UniqueName: \"kubernetes.io/projected/2d928738-b657-4333-b311-13232a0a4aa5-kube-api-access-s6kn2\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:57.746944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.746942    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:57.746944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.746953    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d928738-b657-4333-b311-13232a0a4aa5-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:57.747449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.746961    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d928738-b657-4333-b311-13232a0a4aa5-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:57.769896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.769861    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:57.948270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:57.918533    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:58.032221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:58.029262    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"]
Nov 03 20:22:58.039804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:58.039768    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ff9f5/auth-test-model-9ff9f5-kserve-5b6454dc6f-snzz5"]
Nov 03 20:22:58.076221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:58.073993    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:58.096221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:58.090592    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:22:58.096221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:58.092438    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:22:58.149883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:58.149841    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:22:58.859242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:58.858730    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:22:59.468296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:59.468262    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d928738-b657-4333-b311-13232a0a4aa5" path="/var/lib/kubelet/pods/2d928738-b657-4333-b311-13232a0a4aa5/volumes"
Nov 03 20:22:59.500266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:22:59.500236    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:00.244067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:00.244033    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:00.875761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:00.875712    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:01.519361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:01.519308    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:01.633432 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:01.633393    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:23:02.179173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:02.179142    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:02.664509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:02.664471    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r" podUID="4cbdbcd5-4b29-425c-a0af-e0165833c2c1" containerName="payload-processing" containerID="cri-o://59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633" gracePeriod=25
Nov 03 20:23:02.682816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:02.682634    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65" podUID="bf9a923f-9f14-4691-89a9-5724b7e49c9b" containerName="payload-pre-processing" containerID="cri-o://e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b" gracePeriod=25
Nov 03 20:23:02.895521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:02.895491    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:03.058477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.058450    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:23:03.159043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.159005    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:23:03.175342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.175311    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-kube-api-access-tkhxs\" (UniqueName: \"kubernetes.io/projected/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-kube-api-access-tkhxs\") pod \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") "
Nov 03 20:23:03.175609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.175592    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-plugins-config-volume\") pod \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\" (UID: \"4cbdbcd5-4b29-425c-a0af-e0165833c2c1\") "
Nov 03 20:23:03.176058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.176028    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-plugins-config-volume" pod "4cbdbcd5-4b29-425c-a0af-e0165833c2c1" (UID: "4cbdbcd5-4b29-425c-a0af-e0165833c2c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:03.176444 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.176417    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:03.178111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.178077    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-kube-api-access-tkhxs" pod "4cbdbcd5-4b29-425c-a0af-e0165833c2c1" (UID: "4cbdbcd5-4b29-425c-a0af-e0165833c2c1"). InnerVolumeSpecName "kube-api-access-tkhxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:03.278095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.278015    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9a923f-9f14-4691-89a9-5724b7e49c9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9a923f-9f14-4691-89a9-5724b7e49c9b-plugins-config-volume\") pod \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") "
Nov 03 20:23:03.278276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.278115    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9a923f-9f14-4691-89a9-5724b7e49c9b-kube-api-access-tlqw2\" (UniqueName: \"kubernetes.io/projected/bf9a923f-9f14-4691-89a9-5724b7e49c9b-kube-api-access-tlqw2\") pod \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\" (UID: \"bf9a923f-9f14-4691-89a9-5724b7e49c9b\") "
Nov 03 20:23:03.278393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.278370    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9a923f-9f14-4691-89a9-5724b7e49c9b-plugins-config-volume" pod "bf9a923f-9f14-4691-89a9-5724b7e49c9b" (UID: "bf9a923f-9f14-4691-89a9-5724b7e49c9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:03.278648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.278634    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf9a923f-9f14-4691-89a9-5724b7e49c9b-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:03.278686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.278651    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhxs\" (UniqueName: \"kubernetes.io/projected/4cbdbcd5-4b29-425c-a0af-e0165833c2c1-kube-api-access-tkhxs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:03.280298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.280274    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9a923f-9f14-4691-89a9-5724b7e49c9b-kube-api-access-tlqw2" pod "bf9a923f-9f14-4691-89a9-5724b7e49c9b" (UID: "bf9a923f-9f14-4691-89a9-5724b7e49c9b"). InnerVolumeSpecName "kube-api-access-tlqw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:03.379799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.379765    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlqw2\" (UniqueName: \"kubernetes.io/projected/bf9a923f-9f14-4691-89a9-5724b7e49c9b-kube-api-access-tlqw2\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:03.568272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.559939    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:03.675055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.675022    5282 generic.go:356] "Generic (PLEG): container finished" podID="4cbdbcd5-4b29-425c-a0af-e0165833c2c1" containerID="59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633" exitCode=0
Nov 03 20:23:03.675611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.675085    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"
Nov 03 20:23:03.675611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.675112    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r" event={"ID":"4cbdbcd5-4b29-425c-a0af-e0165833c2c1","Type":"ContainerDied","Data":"59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633"}
Nov 03 20:23:03.675611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.675170    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r" event={"ID":"4cbdbcd5-4b29-425c-a0af-e0165833c2c1","Type":"ContainerDied","Data":"0306838fd85ae5ff603adc43d012e35d4dde846b4c734722039726e7595a20a5"}
Nov 03 20:23:03.675611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.675217    5282 scope.go:122] "RemoveContainer" containerID="59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633"
Nov 03 20:23:03.677077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.677050    5282 generic.go:356] "Generic (PLEG): container finished" podID="bf9a923f-9f14-4691-89a9-5724b7e49c9b" containerID="e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b" exitCode=0
Nov 03 20:23:03.677271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.677086    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"
Nov 03 20:23:03.677458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.677088    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65" event={"ID":"bf9a923f-9f14-4691-89a9-5724b7e49c9b","Type":"ContainerDied","Data":"e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b"}
Nov 03 20:23:03.677566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.677499    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65" event={"ID":"bf9a923f-9f14-4691-89a9-5724b7e49c9b","Type":"ContainerDied","Data":"093f31ef8069480e423822fd54414017fd6ef858d2f5df65efbb7a6341102312"}
Nov 03 20:23:03.696745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.696722    5282 scope.go:122] "RemoveContainer" containerID="59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633"
Nov 03 20:23:03.697022 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:03.696985    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633\": container with ID starting with 59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633 not found: ID does not exist" containerID="59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633"
Nov 03 20:23:03.697077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.697028    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633"} err="failed to get container status \"59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633\": rpc error: code = NotFound desc = could not find container \"59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633\": container with ID starting with 59c27ff7add45db044f8f645582885f468d1af49d63c2a2f6853eb88945dc633 not found: ID does not exist"
Nov 03 20:23:03.697077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.697044    5282 scope.go:122] "RemoveContainer" containerID="e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b"
Nov 03 20:23:03.709398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.709289    5282 scope.go:122] "RemoveContainer" containerID="e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b"
Nov 03 20:23:03.709515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.709408    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"]
Nov 03 20:23:03.709708 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:03.709678    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b\": container with ID starting with e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b not found: ID does not exist" containerID="e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b"
Nov 03 20:23:03.709812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.709716    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b"} err="failed to get container status \"e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b\": rpc error: code = NotFound desc = could not find container \"e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b\": container with ID starting with e5950616f830642f27f2a66fe71c36b797ee5c85b0f246c47c93b33d7b53bb0b not found: ID does not exist"
Nov 03 20:23:03.713924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.713904    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ff9f5-56d4864bf5-5qj65"]
Nov 03 20:23:03.722236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.722187    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"]
Nov 03 20:23:03.727169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:03.727151    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ff9f5-5487f6b6c6-6bj5r"]
Nov 03 20:23:04.279540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:04.279503    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:04.788397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:04.788355    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:04.939750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:04.939672    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:05.061278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:05.060672    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:05.455784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:05.455702    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:05.457132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:05.457110    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbdbcd5-4b29-425c-a0af-e0165833c2c1" path="/var/lib/kubelet/pods/4cbdbcd5-4b29-425c-a0af-e0165833c2c1/volumes"
Nov 03 20:23:05.457623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:05.457609    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9a923f-9f14-4691-89a9-5724b7e49c9b" path="/var/lib/kubelet/pods/bf9a923f-9f14-4691-89a9-5724b7e49c9b/volumes"
Nov 03 20:23:06.033677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:06.033640    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:06.597765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:06.597726    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:07.216522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:07.216495    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:07.900812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:07.900782    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:08.682228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:08.682182    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:09.278520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:09.278477    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:10.022829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:10.022797    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:10.703966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:10.703934    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:11.345230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:11.345184    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:11.515250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:11.511975    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:11.588287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:11.588185    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"]
Nov 03 20:23:11.588788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:11.588754    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy" containerID="cri-o://a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1" gracePeriod=30
Nov 03 20:23:11.643921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:11.643830    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:12.492638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:12.492602    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:13.228797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:13.228760    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:14.004312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:14.004268    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:14.597828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:14.597792    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:15.257698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:15.257662    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:15.976264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:15.976235    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:16.595894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:16.595860    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:16.888074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:16.888048    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:23:17.001627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001585    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-certs\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.001804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001649    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-credential-socket\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.001804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001679    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-podinfo\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.001804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001783    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-socket\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.001934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001846    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-certs" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:17.001934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001851    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-token\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-token\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.002005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001955    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9f6315-7999-43a5-8ca3-6434b1058d58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9f6315-7999-43a5-8ca3-6434b1058d58-istiod-ca-cert\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.002005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.001974    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-credential-socket" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:17.002081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002068    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-socket" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:17.002186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002165    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-data\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.002350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002241    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-kube-api-access-qk28b\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-kube-api-access-qk28b\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.002350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002293    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-envoy\") pod \"4f9f6315-7999-43a5-8ca3-6434b1058d58\" (UID: \"4f9f6315-7999-43a5-8ca3-6434b1058d58\") "
Nov 03 20:23:17.002350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002298    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9f6315-7999-43a5-8ca3-6434b1058d58-istiod-ca-cert" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:17.002526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002429    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-data" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:17.002938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002917    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.002938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002938    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9f6315-7999-43a5-8ca3-6434b1058d58-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.003110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002947    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.003110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002956    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.003110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.002965    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.004369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.004348    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-kube-api-access-qk28b" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "kube-api-access-qk28b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:17.004537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.004514    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-podinfo" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:17.004619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.004566    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-envoy" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:17.004619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.004603    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-token" pod "4f9f6315-7999-43a5-8ca3-6434b1058d58" (UID: "4f9f6315-7999-43a5-8ca3-6434b1058d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:17.104401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.104354    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk28b\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-kube-api-access-qk28b\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.104401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.104397    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.104401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.104409    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.104401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.104419    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9f6315-7999-43a5-8ca3-6434b1058d58-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:17.120038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.119953    5282 generic.go:356] "Generic (PLEG): container finished" podID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerID="a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1" exitCode=0
Nov 03 20:23:17.120038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.119984    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" event={"ID":"4f9f6315-7999-43a5-8ca3-6434b1058d58","Type":"ContainerDied","Data":"a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1"}
Nov 03 20:23:17.120038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.120007    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"
Nov 03 20:23:17.120345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.120043    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb" event={"ID":"4f9f6315-7999-43a5-8ca3-6434b1058d58","Type":"ContainerDied","Data":"be84b97f80fe3c074b0a3793628f7096de171481e2b0ea02981a8a05775bdf15"}
Nov 03 20:23:17.120345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.120070    5282 scope.go:122] "RemoveContainer" containerID="a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1"
Nov 03 20:23:17.132706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.132686    5282 scope.go:122] "RemoveContainer" containerID="046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6"
Nov 03 20:23:17.142891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.142857    5282 scope.go:122] "RemoveContainer" containerID="a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1"
Nov 03 20:23:17.143163 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:17.143145    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1\": container with ID starting with a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1 not found: ID does not exist" containerID="a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1"
Nov 03 20:23:17.143217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.143172    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1"} err="failed to get container status \"a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1\": rpc error: code = NotFound desc = could not find container \"a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1\": container with ID starting with a975c356002c504d1d7288c40448c09b3213ace0212aabbcd289967a2851aba1 not found: ID does not exist"
Nov 03 20:23:17.143217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.143188    5282 scope.go:122] "RemoveContainer" containerID="046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6"
Nov 03 20:23:17.143416 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:17.143401    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6\": container with ID starting with 046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6 not found: ID does not exist" containerID="046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6"
Nov 03 20:23:17.143456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.143419    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6"} err="failed to get container status \"046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6\": rpc error: code = NotFound desc = could not find container \"046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6\": container with ID starting with 046247c93db2a72c35a97f2afe699e2c3d940fc61dfadcf37883236499b0eab6 not found: ID does not exist"
Nov 03 20:23:17.150759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.150736    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"]
Nov 03 20:23:17.154985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.154961    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-9ff9f5-openshift-default-6d6457fb7-g4bfb"]
Nov 03 20:23:17.195963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.195924    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:17.457679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:17.457603    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" path="/var/lib/kubelet/pods/4f9f6315-7999-43a5-8ca3-6434b1058d58/volumes"
Nov 03 20:23:18.083461 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:18.083413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:19.654224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.653882    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:19.688418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.688368    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p" podUID="2ea21d06-e50b-47f7-9921-fe38f1c010d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.175:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:19.730010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.729963    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:23:19.870397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.870342    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:19.911407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.911296    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:19.955394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.955340    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:19.956127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.955657    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:19.962647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:19.962611    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z" podUID="76eaf966-caf0-4394-9f02-488f93755253" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:20.068335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:20.068285    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" podUID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": context deadline exceeded"
Nov 03 20:23:20.249223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:20.245288    5282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:23:20.833541 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:20.824032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:21.459435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:21.459401    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:21.777858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:21.777816    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:22.194323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:22.194238    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:22.291336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:22.282942    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"]
Nov 03 20:23:22.467805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:22.467717    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:22.647224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:22.628316    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:23.211486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:23.211449    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:23.885815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:23.885782    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:24.560741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:24.560702    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:25.276327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:25.276292    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:25.899294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:25.899258    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:26.023813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:26.023779    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"]
Nov 03 20:23:26.028064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:26.028031    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590a217-9f2jb"]
Nov 03 20:23:26.031703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:26.031681    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"]
Nov 03 20:23:26.035173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:26.035147    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e92a75f2-vb6cs"]
Nov 03 20:23:26.704026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:26.703979    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:27.292077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.292010    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww" podUID="44b0b5b7-5b2a-4346-9f41-466db1e3de8f" containerName="payload-pre-processing" containerID="cri-o://36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4" gracePeriod=25
Nov 03 20:23:27.321302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.321276    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:27.460578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.460540    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d0a919-e7ef-480f-b9e7-523f5970b16d" path="/var/lib/kubelet/pods/42d0a919-e7ef-480f-b9e7-523f5970b16d/volumes"
Nov 03 20:23:27.461366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.461342    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fdfebc-775b-49c1-b020-d0cb71b7e4dd" path="/var/lib/kubelet/pods/d5fdfebc-775b-49c1-b020-d0cb71b7e4dd/volumes"
Nov 03 20:23:27.598331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.598306    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:23:27.640994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.640961    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-plugins-config-volume\") pod \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") "
Nov 03 20:23:27.641188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.641134    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-kube-api-access-6nggk\" (UniqueName: \"kubernetes.io/projected/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-kube-api-access-6nggk\") pod \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\" (UID: \"44b0b5b7-5b2a-4346-9f41-466db1e3de8f\") "
Nov 03 20:23:27.641441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.641410    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-plugins-config-volume" pod "44b0b5b7-5b2a-4346-9f41-466db1e3de8f" (UID: "44b0b5b7-5b2a-4346-9f41-466db1e3de8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:27.641983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.641958    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:27.643367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.643344    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-kube-api-access-6nggk" pod "44b0b5b7-5b2a-4346-9f41-466db1e3de8f" (UID: "44b0b5b7-5b2a-4346-9f41-466db1e3de8f"). InnerVolumeSpecName "kube-api-access-6nggk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:27.743557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:27.743524    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nggk\" (UniqueName: \"kubernetes.io/projected/44b0b5b7-5b2a-4346-9f41-466db1e3de8f-kube-api-access-6nggk\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:28.016499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.016457    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:28.431852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.431767    5282 generic.go:356] "Generic (PLEG): container finished" podID="44b0b5b7-5b2a-4346-9f41-466db1e3de8f" containerID="36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4" exitCode=0
Nov 03 20:23:28.431852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.431837    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"
Nov 03 20:23:28.432346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.431845    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww" event={"ID":"44b0b5b7-5b2a-4346-9f41-466db1e3de8f","Type":"ContainerDied","Data":"36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4"}
Nov 03 20:23:28.432346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.431933    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww" event={"ID":"44b0b5b7-5b2a-4346-9f41-466db1e3de8f","Type":"ContainerDied","Data":"2933a4fba066336d5135c6acee6447963d66d6e4734c28dca052ddb6285b346a"}
Nov 03 20:23:28.432346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.431960    5282 scope.go:122] "RemoveContainer" containerID="36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4"
Nov 03 20:23:28.447476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.447445    5282 scope.go:122] "RemoveContainer" containerID="36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4"
Nov 03 20:23:28.447783 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:28.447762    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4\": container with ID starting with 36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4 not found: ID does not exist" containerID="36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4"
Nov 03 20:23:28.447843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.447792    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4"} err="failed to get container status \"36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4\": rpc error: code = NotFound desc = could not find container \"36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4\": container with ID starting with 36d5fcc7b129683aa95c3e9ed6ca74b2f0d357262accbf59c1e64aa009a8ccd4 not found: ID does not exist"
Nov 03 20:23:28.461875 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.461843    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"]
Nov 03 20:23:28.466031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.466004    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-a8e19e-669f4bf495-z4hww"]
Nov 03 20:23:28.710351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:28.710268    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:29.397521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:29.397489    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:29.516622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:29.516584    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b0b5b7-5b2a-4346-9f41-466db1e3de8f" path="/var/lib/kubelet/pods/44b0b5b7-5b2a-4346-9f41-466db1e3de8f/volumes"
Nov 03 20:23:30.066870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:30.066839    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:30.680416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:30.680377    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:31.206960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:31.206926    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:31.357220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:31.353648    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:31.491220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:31.488838    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:31.710816 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:31.710783    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:23:31.758071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:31.757977    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:31.873434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:31.873397    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:32.643614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:32.643582    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:33.248593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:33.248541    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:34.018834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:34.018802    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:34.749605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:34.749570    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:35.490924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:35.490886    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:36.107820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:36.107771    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:36.790063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:36.790020    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:37.023171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:37.023135    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"]
Nov 03 20:23:37.026962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:37.026933    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70dd62a7-m2h92"]
Nov 03 20:23:37.457680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:37.457645    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ee105b-78e0-4c9c-8923-29bb0d2aa102" path="/var/lib/kubelet/pods/c3ee105b-78e0-4c9c-8923-29bb0d2aa102/volumes"
Nov 03 20:23:37.517977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:37.517943    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:37.909680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:37.909646    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:38.027023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:38.026993    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:38.144714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:38.144675    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"]
Nov 03 20:23:38.145210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:38.145162    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" containerID="cri-o://83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b" gracePeriod=30
Nov 03 20:23:38.204470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:38.204391    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:38.618893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:38.618855    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:38.736420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:38.736391    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:39.891223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:39.891178    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:40.697131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:40.697100    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:41.404987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:41.404949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:42.027930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:42.027892    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:42.788324 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:42.788290    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:43.100926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.100842    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:23:43.101093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.100912    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:23:43.458840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.458812    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:23:43.498295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.498257    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:43.600093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.600045    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istiod-ca-cert\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.600763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.600123    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-socket\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.600763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.600516    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-socket" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:43.602009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.601654    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istiod-ca-cert" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:43.610375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.610337    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-kube-api-access-tt56s\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-kube-api-access-tt56s\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.610669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.610640    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-credential-socket\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.610780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.610732    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-podinfo\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.610857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.610842    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-certs\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.610990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.610977    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-token\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-token\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.611045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.611033    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-data\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.611085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.611065    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-envoy\") pod \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\" (UID: \"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a\") "
Nov 03 20:23:43.612150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.611695    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-data" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:43.612150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.611734    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-certs" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:43.612150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.611928    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-credential-socket" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:43.612730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.612467    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.612730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.612497    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.612730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.612513    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.612730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.612528    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.612730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.612542    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.618633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.613705    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"]
Nov 03 20:23:43.625815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.625777    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"]
Nov 03 20:23:43.626776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.626742    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-envoy" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:43.634432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.634381    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-podinfo" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:43.634568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.634423    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-kube-api-access-tt56s" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "kube-api-access-tt56s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:43.634568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.634462    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-token" pod "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" (UID: "e4ab086d-3691-4f4b-a1d2-0e7fb21c063a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:43.713830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.713745    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt56s\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-kube-api-access-tt56s\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.713830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.713776    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.713830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.713787    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.713830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.713809    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:43.748646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.748573    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:43.872951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.872909    5282 generic.go:356] "Generic (PLEG): container finished" podID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerID="83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b" exitCode=0
Nov 03 20:23:43.873422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.872967    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"
Nov 03 20:23:43.873422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.873013    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" event={"ID":"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a","Type":"ContainerDied","Data":"83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b"}
Nov 03 20:23:43.873422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.873083    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d" event={"ID":"e4ab086d-3691-4f4b-a1d2-0e7fb21c063a","Type":"ContainerDied","Data":"a70d6d9c9349a11acc7a5a436ccf0d8f44be691abe3c8ba9224258573ed30200"}
Nov 03 20:23:43.873422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.873113    5282 scope.go:122] "RemoveContainer" containerID="83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b"
Nov 03 20:23:43.885561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.885544    5282 scope.go:122] "RemoveContainer" containerID="dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd"
Nov 03 20:23:43.896234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.896210    5282 scope.go:122] "RemoveContainer" containerID="83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b"
Nov 03 20:23:43.896535 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:43.896515    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b\": container with ID starting with 83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b not found: ID does not exist" containerID="83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b"
Nov 03 20:23:43.896588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.896543    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b"} err="failed to get container status \"83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b\": rpc error: code = NotFound desc = could not find container \"83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b\": container with ID starting with 83a109a2c9ad0056b4a946af842921b328264fe55bbfeef54780920a3b97437b not found: ID does not exist"
Nov 03 20:23:43.896588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.896562    5282 scope.go:122] "RemoveContainer" containerID="dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd"
Nov 03 20:23:43.896873 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:43.896841    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd\": container with ID starting with dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd not found: ID does not exist" containerID="dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd"
Nov 03 20:23:43.897004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.896908    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd"} err="failed to get container status \"dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd\": rpc error: code = NotFound desc = could not find container \"dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd\": container with ID starting with dc7cfa47f57043eaf1b9aa5567dfec544038c623ade67a00723be31f982b82bd not found: ID does not exist"
Nov 03 20:23:43.897980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.897960    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"]
Nov 03 20:23:43.901909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:43.901889    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a8e19e-openshift-default-5d9485bf-hp48d"]
Nov 03 20:23:44.372076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:44.372046    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:45.079362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:45.079328    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:45.457329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:45.457254    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" path="/var/lib/kubelet/pods/e4ab086d-3691-4f4b-a1d2-0e7fb21c063a/volumes"
Nov 03 20:23:45.466851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:45.466823    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:23:45.801966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:45.801928    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:46.605656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:46.605624    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:47.457350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:47.457185    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:48.021353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:48.021308    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"]
Nov 03 20:23:48.025125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:48.025098    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0bc615-zskqt"]
Nov 03 20:23:48.087613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:48.087572    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:48.801710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:48.801674    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:49.090730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.090617    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:49.198597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.198566    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:49.514222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.508095    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fbc971b-8d78-4467-8e2b-a74501f09f6e" path="/var/lib/kubelet/pods/2fbc971b-8d78-4467-8e2b-a74501f09f6e/volumes"
Nov 03 20:23:49.784514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.784435    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"]
Nov 03 20:23:49.785257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785239    5282 cpu_manager.go:414] "containerMap: removing container" podUID="4cbdbcd5-4b29-425c-a0af-e0165833c2c1" containerName="payload-processing"
Nov 03 20:23:49.785325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785259    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbdbcd5-4b29-425c-a0af-e0165833c2c1" containerName="payload-processing"
Nov 03 20:23:49.785325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785276    5282 cpu_manager.go:414] "containerMap: removing container" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy"
Nov 03 20:23:49.785325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785281    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy"
Nov 03 20:23:49.785325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785292    5282 cpu_manager.go:414] "containerMap: removing container" podUID="2d928738-b657-4333-b311-13232a0a4aa5" containerName="main"
Nov 03 20:23:49.785325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785297    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d928738-b657-4333-b311-13232a0a4aa5" containerName="main"
Nov 03 20:23:49.785325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785311    5282 cpu_manager.go:414] "containerMap: removing container" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy"
Nov 03 20:23:49.785325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785316    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785338    5282 cpu_manager.go:414] "containerMap: removing container" podUID="44b0b5b7-5b2a-4346-9f41-466db1e3de8f" containerName="payload-pre-processing"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785343    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b0b5b7-5b2a-4346-9f41-466db1e3de8f" containerName="payload-pre-processing"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785364    5282 cpu_manager.go:414] "containerMap: removing container" podUID="bf9a923f-9f14-4691-89a9-5724b7e49c9b" containerName="payload-pre-processing"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785369    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9a923f-9f14-4691-89a9-5724b7e49c9b" containerName="payload-pre-processing"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785491    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9a923f-9f14-4691-89a9-5724b7e49c9b" containerName="payload-pre-processing"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785503    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbdbcd5-4b29-425c-a0af-e0165833c2c1" containerName="payload-processing"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785511    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785517    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d928738-b657-4333-b311-13232a0a4aa5" containerName="main"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785525    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy"
Nov 03 20:23:49.785641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.785533    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b0b5b7-5b2a-4346-9f41-466db1e3de8f" containerName="payload-pre-processing"
Nov 03 20:23:49.791412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.791389    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.796810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.796787    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b55105-openshift-default-dockercfg-b42xt\""
Nov 03 20:23:49.803501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.803477    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"]
Nov 03 20:23:49.817535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.817458    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:49.950902 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.950860    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.950963    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.951119    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.951162    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.951337    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.951540    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.951694    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5p52\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-kube-api-access-v5p52\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.951747    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:49.951818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:49.951786    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.053485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053398    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.053485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053458    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.053763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053489    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.053763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053542    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.053763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053642    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.053763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053758    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5p52\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-kube-api-access-v5p52\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053783    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053811    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.053921    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.054050    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-certs\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.054185    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-credential-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.054302    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-data\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.054345    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-socket\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.054696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.054669    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istiod-ca-cert\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.056619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.056597    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-envoy\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.057059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.057038    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-podinfo\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.061713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.061684    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-token\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.062112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.062089    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5p52\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-kube-api-access-v5p52\") pod \"e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") " pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.110794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.110765    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:50.403424 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.403391    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"]
Nov 03 20:23:50.404389 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:23:50.404363    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb253246c_0adb_4d4d_b6db_4bfa0c3993f4.slice/crio-0bb84febe257b3b097a303f4f6f0e02680396eba2cc5938bc43840e911323e17 WatchSource:0}: Error finding container 0bb84febe257b3b097a303f4f6f0e02680396eba2cc5938bc43840e911323e17: Status 404 returned error can't find the container with id 0bb84febe257b3b097a303f4f6f0e02680396eba2cc5938bc43840e911323e17
Nov 03 20:23:50.412049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.412003    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:23:50.412141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.412101    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:23:50.412189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.412151    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:23:50.613877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.613842    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:50.983538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.983501    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" event={"ID":"b253246c-0adb-4d4d-b6db-4bfa0c3993f4","Type":"ContainerStarted","Data":"a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664"}
Nov 03 20:23:50.983729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:50.983582    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" event={"ID":"b253246c-0adb-4d4d-b6db-4bfa0c3993f4","Type":"ContainerStarted","Data":"0bb84febe257b3b097a303f4f6f0e02680396eba2cc5938bc43840e911323e17"}
Nov 03 20:23:51.008875 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:51.008819    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podStartSLOduration=2.008804505 podStartE2EDuration="2.008804505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:51.004957191 +0000 UTC m=+2623.674158662" watchObservedRunningTime="2026-11-03 20:23:51.008804505 +0000 UTC m=+2623.678005975"
Nov 03 20:23:51.111052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:51.111015    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:51.113122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:51.113092    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:23:51.113283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:51.113141    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:23:51.211958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:51.211903    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:51.899683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:51.899639    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:52.111685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:52.111649    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:23:52.112084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:52.111724    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:23:52.500314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:52.500282    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:53.111870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:53.111827    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:23:53.112410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:53.111882    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:23:53.197217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:53.196636    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:53.818079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:53.818032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:54.111854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:54.111758    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:23:54.111854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:54.111819    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:23:54.799221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:54.798399    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:55.111833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:55.111732    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:23:55.111833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:55.111806    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:23:55.404691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:55.404602    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:56.106246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:56.106180    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:56.114221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:56.111651    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:23:56.114221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:56.111759    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:23:56.814327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:56.814290    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:57.112036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:57.111954    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:23:57.112192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:57.112043    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 20:23:57.504162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:57.504123    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:58.110636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.110604    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:58.116641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.116616    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:58.117078 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.117061    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:58.117758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.117742    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:23:58.657081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.657036    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" containerID="cri-o://36cb5ac553e0182881017cd7c6ec7be4d1cd386ee2cf0c4234a5287a4ce3538a" gracePeriod=45
Nov 03 20:23:58.658296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.658250    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" containerID="cri-o://5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5" gracePeriod=45
Nov 03 20:23:58.785965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.785935    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:23:58.870655 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.869896    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused"
Nov 03 20:23:58.955304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:58.954761    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Nov 03 20:23:59.069519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.069486    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:23:59.091010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.090975    5282 generic.go:356] "Generic (PLEG): container finished" podID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerID="36cb5ac553e0182881017cd7c6ec7be4d1cd386ee2cf0c4234a5287a4ce3538a" exitCode=0
Nov 03 20:23:59.091189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.091084    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" event={"ID":"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d","Type":"ContainerDied","Data":"36cb5ac553e0182881017cd7c6ec7be4d1cd386ee2cf0c4234a5287a4ce3538a"}
Nov 03 20:23:59.101721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.093995    5282 generic.go:356] "Generic (PLEG): container finished" podID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerID="5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5" exitCode=0
Nov 03 20:23:59.101721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.095291    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"
Nov 03 20:23:59.101721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.095649    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" event={"ID":"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b","Type":"ContainerDied","Data":"5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5"}
Nov 03 20:23:59.101721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.095686    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7" event={"ID":"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b","Type":"ContainerDied","Data":"6af24c78cc83afdd53ef94e43a84511d879bdf0f46da00f61bf3dbbe0d2bd9d8"}
Nov 03 20:23:59.101721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.095710    5282 scope.go:122] "RemoveContainer" containerID="5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5"
Nov 03 20:23:59.113874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.113850    5282 scope.go:122] "RemoveContainer" containerID="5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5"
Nov 03 20:23:59.114386 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:23:59.114355    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5\": container with ID starting with 5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5 not found: ID does not exist" containerID="5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5"
Nov 03 20:23:59.114457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.114400    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5"} err="failed to get container status \"5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5\": rpc error: code = NotFound desc = could not find container \"5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5\": container with ID starting with 5ad7129b3143e27284a8cae3f5d9d36bdda79838b2b637c251c6e33e44f2edd5 not found: ID does not exist"
Nov 03 20:23:59.115083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.115065    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:23:59.149841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.149801    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tmp-dir\") pod \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") "
Nov 03 20:23:59.150046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.149876    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-dshm\") pod \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") "
Nov 03 20:23:59.150046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.149998    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tls-certs\") pod \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") "
Nov 03 20:23:59.150272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.150249    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-kube-api-access-2fvh5\" (UniqueName: \"kubernetes.io/projected/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-kube-api-access-2fvh5\") pod \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") "
Nov 03 20:23:59.150416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.150373    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-model-cache\") pod \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") "
Nov 03 20:23:59.150486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.150456    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-home\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-home\") pod \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\" (UID: \"9d1a2ac4-29ea-48fe-b139-d33c7b4a261b\") "
Nov 03 20:23:59.151555 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.151527    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-home" pod "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" (UID: "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.152900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.152868    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-dshm" pod "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" (UID: "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.153057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.153023    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tmp-dir" pod "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" (UID: "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.154721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.154698    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tls-certs" pod "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" (UID: "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:59.155016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.154941    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-kube-api-access-2fvh5" pod "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" (UID: "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b"). InnerVolumeSpecName "kube-api-access-2fvh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:59.155182 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.155160    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-model-cache" pod "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" (UID: "9d1a2ac4-29ea-48fe-b139-d33c7b4a261b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.252519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.252465    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-kube-api-access-zf7jw\" (UniqueName: \"kubernetes.io/projected/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-kube-api-access-zf7jw\") pod \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") "
Nov 03 20:23:59.252519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.252538    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-model-cache\") pod \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") "
Nov 03 20:23:59.252958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.252768    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-home\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-home\") pod \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") "
Nov 03 20:23:59.252958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.252819    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tls-certs\") pod \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") "
Nov 03 20:23:59.252958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.252840    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-dshm\") pod \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") "
Nov 03 20:23:59.252958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.252864    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tmp-dir\") pod \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\" (UID: \"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d\") "
Nov 03 20:23:59.254250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.253705    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tmp-dir" pod "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" (UID: "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.254250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.253812    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-model-cache" pod "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" (UID: "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.254250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.253948    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-home" pod "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" (UID: "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.254250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254235    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254257    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254273    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254288    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvh5\" (UniqueName: \"kubernetes.io/projected/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-kube-api-access-2fvh5\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254303    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254316    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254332    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254345    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.254635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.254358    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.256049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.256024    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-dshm" pod "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" (UID: "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:59.256049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.256039    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tls-certs" pod "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" (UID: "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:59.256413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.256392    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-kube-api-access-zf7jw" pod "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" (UID: "57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d"). InnerVolumeSpecName "kube-api-access-zf7jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:59.356237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.356159    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7jw\" (UniqueName: \"kubernetes.io/projected/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-kube-api-access-zf7jw\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.356237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.356239    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.356476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.356252    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:59.430158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.430122    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"]
Nov 03 20:23:59.442752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.442660    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-2-w3-b87733-kserve-5b89b5f567-4h6l7"]
Nov 03 20:23:59.456742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.456702    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" path="/var/lib/kubelet/pods/9d1a2ac4-29ea-48fe-b139-d33c7b4a261b/volumes"
Nov 03 20:23:59.529893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:23:59.529806    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:00.103096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:00.103067    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"
Nov 03 20:24:00.103096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:00.103074    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw" event={"ID":"57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d","Type":"ContainerDied","Data":"dd9168a20399197e1a38abd56b2ebd92ac763ebe286e3abc2783c9087c66b5c9"}
Nov 03 20:24:00.103364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:00.103125    5282 scope.go:122] "RemoveContainer" containerID="36cb5ac553e0182881017cd7c6ec7be4d1cd386ee2cf0c4234a5287a4ce3538a"
Nov 03 20:24:00.122223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:00.122166    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"]
Nov 03 20:24:00.126603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:00.126579    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b87733/distinct-w3-b87733-kserve-658b44f948-bs9hw"]
Nov 03 20:24:00.320807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:00.320698    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:01.042656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:01.034976    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:01.456683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:01.456602    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" path="/var/lib/kubelet/pods/57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d/volumes"
Nov 03 20:24:01.643330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:01.643288    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:01.648679 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:24:01.648640    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:24:02.263244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:02.262997    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:03.041419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:03.041380    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:03.625015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:03.624983    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:04.310072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:04.310038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:04.862161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:04.862122    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:05.506827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:05.506791    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:06.149390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:06.149345    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:06.860933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:06.860827    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:07.377298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:07.377263    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:07.977401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:07.977365    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:08.616375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:08.616327    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:09.293035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:09.293002    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:09.882153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:09.882072    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:10.691980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:10.691952    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:11.232944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:11.232906    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:11.597887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:11.597779    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:11.781348 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:11.781309    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:11.922613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:11.922531    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:12.709412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:12.709374    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:13.168151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:13.168119    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:13.446193 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:13.446109    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:14.093482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:14.093446    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:14.833230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:14.831334    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:15.487583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:15.487549    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:16.028329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:16.028293    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"]
Nov 03 20:24:16.035343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:16.035307    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c8f4714-ln9br"]
Nov 03 20:24:16.113523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:16.113488    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:16.802041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:16.802005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:17.528344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:17.528306    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8cf104b-b5ba-4ed1-a50d-65f867fc69cb" path="/var/lib/kubelet/pods/c8cf104b-b5ba-4ed1-a50d-65f867fc69cb/volumes"
Nov 03 20:24:17.645839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:17.645804    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:17.999026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:17.998993    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:18.811443 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:18.811413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:19.102272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.099390    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:19.102272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.102247    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"]
Nov 03 20:24:19.139220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.135770    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ad4dd4c-r8b7b"]
Nov 03 20:24:19.296036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.295995    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:19.329329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.329270    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"]
Nov 03 20:24:19.330291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.330258    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy" containerID="cri-o://9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9" gracePeriod=30
Nov 03 20:24:19.432591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.432499    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:19.460987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:19.460952    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10273c46-7990-4f1d-8d7b-864cd61ef0b9" path="/var/lib/kubelet/pods/10273c46-7990-4f1d-8d7b-864cd61ef0b9/volumes"
Nov 03 20:24:20.011339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:20.011307    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:20.221651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:20.221622    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:20.353824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:20.353746    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:20.504224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:20.496448    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:20.897943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:20.897908    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:21.546563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:21.546531    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:21.691650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:21.691613    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:21.863359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:21.863271    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:22.475975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:22.475940    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:23.282869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:23.282837    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:23.886527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:23.886495    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:24.658039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.657949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:24.679480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.673479    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:24:24.780599 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.780558    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-podinfo\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.780793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.780664    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-socket\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.780793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.780703    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-credential-socket\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.780793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.780737    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-kube-api-access-kld42\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-kube-api-access-kld42\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.780996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.780838    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-data\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.780996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.780988    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-certs\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.781187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.781041    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-envoy\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.781187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.781057    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-socket" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.781187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.781136    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e98860-af8e-49cb-be50-efe9bb8a8a27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e98860-af8e-49cb-be50-efe9bb8a8a27-istiod-ca-cert\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.781187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.781177    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-token\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-token\") pod \"93e98860-af8e-49cb-be50-efe9bb8a8a27\" (UID: \"93e98860-af8e-49cb-be50-efe9bb8a8a27\") "
Nov 03 20:24:24.781433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.781309    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-data" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.781710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.781671    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e98860-af8e-49cb-be50-efe9bb8a8a27-istiod-ca-cert" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:24.781870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.781849    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-credential-socket" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.782125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.782106    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-certs" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.782720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.782695    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.782837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.782727    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e98860-af8e-49cb-be50-efe9bb8a8a27-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.782837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.782744    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.782837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.782760    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.782837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.782775    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.783496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.783458    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-podinfo" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:24:24.783953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.783932    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-envoy" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:24.784179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.784154    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-token" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:24.786724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.786699    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-kube-api-access-kld42" pod "93e98860-af8e-49cb-be50-efe9bb8a8a27" (UID: "93e98860-af8e-49cb-be50-efe9bb8a8a27"). InnerVolumeSpecName "kube-api-access-kld42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:24.884673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.884629    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.884673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.884672    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.884883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.884689    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e98860-af8e-49cb-be50-efe9bb8a8a27-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:24.884883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:24.884704    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kld42\" (UniqueName: \"kubernetes.io/projected/93e98860-af8e-49cb-be50-efe9bb8a8a27-kube-api-access-kld42\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:25.379412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.379377    5282 generic.go:356] "Generic (PLEG): container finished" podID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerID="9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9" exitCode=0
Nov 03 20:24:25.379611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.379441    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"
Nov 03 20:24:25.379611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.379455    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" event={"ID":"93e98860-af8e-49cb-be50-efe9bb8a8a27","Type":"ContainerDied","Data":"9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9"}
Nov 03 20:24:25.379611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.379510    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w" event={"ID":"93e98860-af8e-49cb-be50-efe9bb8a8a27","Type":"ContainerDied","Data":"e602eee5cada232aee2ec5c4203011830f30addfff58c1497ceb3eeee5bc91d5"}
Nov 03 20:24:25.379611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.379538    5282 scope.go:122] "RemoveContainer" containerID="9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9"
Nov 03 20:24:25.391657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.391638    5282 scope.go:122] "RemoveContainer" containerID="9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9"
Nov 03 20:24:25.391982 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:24:25.391959    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9\": container with ID starting with 9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9 not found: ID does not exist" containerID="9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9"
Nov 03 20:24:25.392055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.391995    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9"} err="failed to get container status \"9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9\": rpc error: code = NotFound desc = could not find container \"9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9\": container with ID starting with 9764d3cf0ed458c40c511129afbfa7ce50e34d38591c69683ee636237be524d9 not found: ID does not exist"
Nov 03 20:24:25.405868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.405542    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"]
Nov 03 20:24:25.410910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.410886    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b87733-openshift-default-989c6f4f5-f4t8w"]
Nov 03 20:24:25.457954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.457925    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" path="/var/lib/kubelet/pods/93e98860-af8e-49cb-be50-efe9bb8a8a27/volumes"
Nov 03 20:24:25.568464 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:25.568409    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:26.396426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:26.396392    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:27.125164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:27.125122    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:28.446758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:28.446723    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:28.578849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:28.576566    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:29.297181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:29.297141    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:30.077943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:30.077910    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:30.679886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:30.679832    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:31.289029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:31.288994    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:31.615395 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:24:31.615320    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:24:31.904563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:31.904469    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:32.556453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:32.556417    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:33.195849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:33.195789    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:33.806368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:33.806337    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:34.406765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:34.406733    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:35.089053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:35.089008    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:35.786686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:35.786603    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:37.202494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:37.202461    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:37.804510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:37.804477    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:38.391778 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:38.391748    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:39.004636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:39.004605    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:39.682269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:39.682232    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:40.294873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:40.294820    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:40.882911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:40.882881    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:41.522824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:41.522792    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:42.207346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:42.207312    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:43.103046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:43.103006    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:43.787527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:43.787496    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:44.427598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:44.427553    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:45.095776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:45.095716    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:45.697076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:45.697038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:46.295429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:46.295397    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:46.972951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:46.972917    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:47.607176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:47.607137    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:48.208323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:48.208282    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:48.802236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:48.802156    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:49.482723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:49.482690    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:50.074514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:50.074469    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:50.721302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:50.721267    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:51.312751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:51.312703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:51.927681 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:51.927646    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:52.594569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:52.594534    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:53.299417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:53.299359    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:53.994972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:53.994934    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:54.726376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:54.726331    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:55.495136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:55.471183    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:56.104813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:56.104778    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:56.724660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:56.724629    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:57.361742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:57.361702    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:58.003426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:58.003396    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:58.672765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:58.672725    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:59.273378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.273335    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"]
Nov 03 20:24:59.274526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274507    5282 cpu_manager.go:414] "containerMap: removing container" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy"
Nov 03 20:24:59.274587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274528    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy"
Nov 03 20:24:59.274587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274552    5282 cpu_manager.go:414] "containerMap: removing container" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy"
Nov 03 20:24:59.274587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274560    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy"
Nov 03 20:24:59.274587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274580    5282 cpu_manager.go:414] "containerMap: removing container" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy"
Nov 03 20:24:59.274718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274589    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy"
Nov 03 20:24:59.274718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274639    5282 cpu_manager.go:414] "containerMap: removing container" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main"
Nov 03 20:24:59.274718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274647    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main"
Nov 03 20:24:59.274718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274665    5282 cpu_manager.go:414] "containerMap: removing container" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main"
Nov 03 20:24:59.274718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274673    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main"
Nov 03 20:24:59.274869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274835    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e98860-af8e-49cb-be50-efe9bb8a8a27" containerName="istio-proxy"
Nov 03 20:24:59.274869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274850    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ab086d-3691-4f4b-a1d2-0e7fb21c063a" containerName="istio-proxy"
Nov 03 20:24:59.274935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274868    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ffb279-2c83-4ad1-aa9e-f3b1ed46fe9d" containerName="main"
Nov 03 20:24:59.274935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274879    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1a2ac4-29ea-48fe-b139-d33c7b4a261b" containerName="main"
Nov 03 20:24:59.274935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.274893    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9f6315-7999-43a5-8ca3-6434b1058d58" containerName="istio-proxy"
Nov 03 20:24:59.279702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.279672    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.285536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.285510    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-456f55-openshift-default-dockercfg-8jtw5\""
Nov 03 20:24:59.291069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.291040    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"]
Nov 03 20:24:59.393284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.393239    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:24:59.459631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.459586    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94gsj\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-kube-api-access-94gsj\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.459814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.459718    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istiod-ca-cert\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.459814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.459772    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-data\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.459928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.459814    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.459980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.459956    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-certs\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.460031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.460000    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-podinfo\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.460083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.460031    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-token\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.460246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.460222    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-envoy\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.460438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.460412    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-credential-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562267    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-credential-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562350    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94gsj\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-kube-api-access-94gsj\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562403    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istiod-ca-cert\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562585    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-data\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562633    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562712    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-certs\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562755    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-podinfo\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562757    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-credential-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562795    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-token\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562894    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-data\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.562925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.562877    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-envoy\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.563217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.563027    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-socket\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.563217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.563086    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-certs\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.563217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.563175    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istiod-ca-cert\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.565150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.565126    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-envoy\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.565386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.565365    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-podinfo\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.570185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.570160    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-token\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.570364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.570344    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94gsj\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-kube-api-access-94gsj\") pod \"e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") " pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.694982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.694949    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:24:59.984696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.984664    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"]
Nov 03 20:24:59.985901 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:24:59.985874    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7652f6b_b3d5_4aa5_a49e_ab25fc8f65b3.slice/crio-05a59703c5b9202948a8b997a6e442465a909c73fa5f5e3cb546287f3fc02193 WatchSource:0}: Error finding container 05a59703c5b9202948a8b997a6e442465a909c73fa5f5e3cb546287f3fc02193: Status 404 returned error can't find the container with id 05a59703c5b9202948a8b997a6e442465a909c73fa5f5e3cb546287f3fc02193
Nov 03 20:24:59.988800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.988727    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:24:59.988889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.988809    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:24:59.988889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.988850    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:24:59.989976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:24:59.989948    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:00.608102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:00.608067    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:00.855843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:00.855800    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" event={"ID":"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3","Type":"ContainerStarted","Data":"f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e"}
Nov 03 20:25:00.856041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:00.855877    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" event={"ID":"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3","Type":"ContainerStarted","Data":"05a59703c5b9202948a8b997a6e442465a909c73fa5f5e3cb546287f3fc02193"}
Nov 03 20:25:00.891280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:00.890116    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podStartSLOduration=1.890090413 podStartE2EDuration="1.890090413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:00.882980142 +0000 UTC m=+2693.552181625" watchObservedRunningTime="2026-11-03 20:25:00.890090413 +0000 UTC m=+2693.559291884"
Nov 03 20:25:01.371092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:01.371060    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:01.630263 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:01.630160    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:25:01.696225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:01.695533    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:25:01.699438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:01.699400    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:25:01.699649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:01.699491    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 20:25:01.998650 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:01.998619    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:02.599929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:02.599863    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:02.700711 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:02.700670    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:25:02.701301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:02.701266    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 20:25:03.211616 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:03.211578    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:03.698763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:03.698723    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:25:03.700395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:03.700351    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 20:25:03.704047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:03.704024    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:25:03.924762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:03.924725    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:04.149964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:04.149921    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"]
Nov 03 20:25:04.211837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:04.211764    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:04.709336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:04.709279    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:25:04.709928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:04.709894    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 20:25:04.860144 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:04.860105    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:05.695426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:05.695388    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:25:05.695728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:05.695695    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 20:25:06.320226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:06.313030    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:06.697882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:06.697788    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:25:06.698147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:06.698116    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 20:25:06.905470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:06.905411    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:07.680630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:07.680571    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:07.716229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:07.714563    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:25:07.720244 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:07.716521    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:25:07.774746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:07.774116    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:25:08.502581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:08.502540    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:09.085986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.085951    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:09.556234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.556187    5282 scope.go:122] "RemoveContainer" containerID="00a50bb2930928670beea3f4dd7e11ff204aad12574751befc5634bcf0f3808c"
Nov 03 20:25:09.567096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.567073    5282 scope.go:122] "RemoveContainer" containerID="2f2ebc4a50e5d639bb2acd516d069b46758ef7dc7c492d251ee70a495104d76a"
Nov 03 20:25:09.577066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.577045    5282 scope.go:122] "RemoveContainer" containerID="583feceaaca1d3d4c8034b9566ff1d255abf4ed93b268a0abbadcc54d5706458"
Nov 03 20:25:09.587047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.587020    5282 scope.go:122] "RemoveContainer" containerID="c2ed107193cdd179ea316b91d8dc501136c472c3a4bd93bcfd39cffd78d2952d"
Nov 03 20:25:09.597054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.597033    5282 scope.go:122] "RemoveContainer" containerID="a5b4d7254dfffc61714c397c44bc0abb23ecc1ff4e89ec8ce84a65c5743e3586"
Nov 03 20:25:09.606672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.606655    5282 scope.go:122] "RemoveContainer" containerID="a4a5bb99ba6702dd3ad9f92a142685564e411416a93e0cda8016b998a76075c1"
Nov 03 20:25:09.621941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.621919    5282 scope.go:122] "RemoveContainer" containerID="a79fce7003093f237e61b81360d8355548ddc52e02163d91954fbf46591749f9"
Nov 03 20:25:09.888712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:09.888630    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:10.792773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:10.792734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:11.457624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:11.457589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:12.088333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:12.088296    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:12.206256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:12.206225    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:12.815130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:12.815099    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:13.604054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:13.604023    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:14.206523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:14.206482    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:14.898591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:14.898550    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:15.156842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:15.156739    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"]
Nov 03 20:25:15.214524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:15.214455    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:15.865758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:15.865343    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:16.596808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:16.596776    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:17.281539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:17.281509    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:18.092694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:18.092659    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:18.882730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:18.882694    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:19.208043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.207917    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx" podUID="b272424b-a840-4c8f-9622-09136854de34" containerName="main" containerID="cri-o://ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62" gracePeriod=45
Nov 03 20:25:19.529044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.529018    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:25:19.616742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.616707    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b272424b-a840-4c8f-9622-09136854de34-tls-certs\" (UniqueName: \"kubernetes.io/secret/b272424b-a840-4c8f-9622-09136854de34-tls-certs\") pod \"b272424b-a840-4c8f-9622-09136854de34\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") "
Nov 03 20:25:19.616940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.616760    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b272424b-a840-4c8f-9622-09136854de34-kube-api-access-ldgcp\" (UniqueName: \"kubernetes.io/projected/b272424b-a840-4c8f-9622-09136854de34-kube-api-access-ldgcp\") pod \"b272424b-a840-4c8f-9622-09136854de34\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") "
Nov 03 20:25:19.616940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.616865    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-dshm\") pod \"b272424b-a840-4c8f-9622-09136854de34\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") "
Nov 03 20:25:19.617077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.616979    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-model-cache\") pod \"b272424b-a840-4c8f-9622-09136854de34\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") "
Nov 03 20:25:19.617077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.617051    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-tmp-dir\") pod \"b272424b-a840-4c8f-9622-09136854de34\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") "
Nov 03 20:25:19.617184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.617165    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-home\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-home\") pod \"b272424b-a840-4c8f-9622-09136854de34\" (UID: \"b272424b-a840-4c8f-9622-09136854de34\") "
Nov 03 20:25:19.617331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.617302    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-model-cache" pod "b272424b-a840-4c8f-9622-09136854de34" (UID: "b272424b-a840-4c8f-9622-09136854de34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:19.617605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.617578    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-tmp-dir" pod "b272424b-a840-4c8f-9622-09136854de34" (UID: "b272424b-a840-4c8f-9622-09136854de34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:19.617732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.617588    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-home" pod "b272424b-a840-4c8f-9622-09136854de34" (UID: "b272424b-a840-4c8f-9622-09136854de34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:19.618042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.618016    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:19.618130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.618051    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:19.618130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.618066    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:19.619634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.619584    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b272424b-a840-4c8f-9622-09136854de34-tls-certs" pod "b272424b-a840-4c8f-9622-09136854de34" (UID: "b272424b-a840-4c8f-9622-09136854de34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:19.619745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.619668    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-dshm" pod "b272424b-a840-4c8f-9622-09136854de34" (UID: "b272424b-a840-4c8f-9622-09136854de34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:19.620248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.620189    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b272424b-a840-4c8f-9622-09136854de34-kube-api-access-ldgcp" pod "b272424b-a840-4c8f-9622-09136854de34" (UID: "b272424b-a840-4c8f-9622-09136854de34"). InnerVolumeSpecName "kube-api-access-ldgcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:19.674056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.674005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:19.728268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.726388    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b272424b-a840-4c8f-9622-09136854de34-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:19.728268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.726427    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldgcp\" (UniqueName: \"kubernetes.io/projected/b272424b-a840-4c8f-9622-09136854de34-kube-api-access-ldgcp\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:19.728268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:19.726447    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b272424b-a840-4c8f-9622-09136854de34-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:20.137445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.137406    5282 generic.go:356] "Generic (PLEG): container finished" podID="b272424b-a840-4c8f-9622-09136854de34" containerID="ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62" exitCode=0
Nov 03 20:25:20.137634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.137453    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"
Nov 03 20:25:20.137634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.137487    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx" event={"ID":"b272424b-a840-4c8f-9622-09136854de34","Type":"ContainerDied","Data":"ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62"}
Nov 03 20:25:20.137634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.137532    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx" event={"ID":"b272424b-a840-4c8f-9622-09136854de34","Type":"ContainerDied","Data":"ea96f836e3c0c54d086ed1f08adac49cd4819c1eabdeca3f4160f656ebad211b"}
Nov 03 20:25:20.137634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.137549    5282 scope.go:122] "RemoveContainer" containerID="ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62"
Nov 03 20:25:20.153233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.153165    5282 scope.go:122] "RemoveContainer" containerID="ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62"
Nov 03 20:25:20.154313 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:20.154281    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62\": container with ID starting with ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62 not found: ID does not exist" containerID="ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62"
Nov 03 20:25:20.154414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.154326    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62"} err="failed to get container status \"ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62\": rpc error: code = NotFound desc = could not find container \"ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62\": container with ID starting with ab9b2519d1b5ffc5d894e64562c9d4a903e46eb25e78b3847286be4164c00c62 not found: ID does not exist"
Nov 03 20:25:20.170396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.170365    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"]
Nov 03 20:25:20.174068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.174046    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-46c34c/test-model-46c34c-kserve-7b557f9477-826dx"]
Nov 03 20:25:20.380948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:20.380914    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:21.196147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:21.196103    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:21.457392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:21.457317    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b272424b-a840-4c8f-9622-09136854de34" path="/var/lib/kubelet/pods/b272424b-a840-4c8f-9622-09136854de34/volumes"
Nov 03 20:25:21.911398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:21.911354    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:22.039592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:22.039558    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:22.730642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:22.730607    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:23.392509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:23.392469    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:24.250076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:24.250044    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:25.567922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:25.567884    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"]
Nov 03 20:25:25.624559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:25.624526    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:25.739956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:25.739923    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:26.493131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:26.493094    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:27.260227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:27.242682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:27.902350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:27.902313    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:28.583323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:28.583289    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:29.122957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:29.122910    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:29.273908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:29.273869    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:29.601847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:29.601806    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:29.845988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:29.845951    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:29.959925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:29.959836    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:30.191513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.191476    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p" podUID="2ea21d06-e50b-47f7-9921-fe38f1c010d9" containerName="main" containerID="cri-o://9954b669d010d457ee5f0b91791039ce8ec5e8032e2e71e3192357f31ed50091" gracePeriod=45
Nov 03 20:25:30.330074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.330043    5282 generic.go:356] "Generic (PLEG): container finished" podID="2ea21d06-e50b-47f7-9921-fe38f1c010d9" containerID="9954b669d010d457ee5f0b91791039ce8ec5e8032e2e71e3192357f31ed50091" exitCode=0
Nov 03 20:25:30.330259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.330122    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p" event={"ID":"2ea21d06-e50b-47f7-9921-fe38f1c010d9","Type":"ContainerDied","Data":"9954b669d010d457ee5f0b91791039ce8ec5e8032e2e71e3192357f31ed50091"}
Nov 03 20:25:30.490390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.490365    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:25:30.525505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525474    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tls-certs\") pod \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") "
Nov 03 20:25:30.525699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525536    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-home\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-home\") pod \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") "
Nov 03 20:25:30.525699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525572    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea21d06-e50b-47f7-9921-fe38f1c010d9-kube-api-access-qfr9j\" (UniqueName: \"kubernetes.io/projected/2ea21d06-e50b-47f7-9921-fe38f1c010d9-kube-api-access-qfr9j\") pod \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") "
Nov 03 20:25:30.525699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525636    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-model-cache\") pod \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") "
Nov 03 20:25:30.525876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525698    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tmp-dir\") pod \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") "
Nov 03 20:25:30.525876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525849    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-dshm\") pod \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\" (UID: \"2ea21d06-e50b-47f7-9921-fe38f1c010d9\") "
Nov 03 20:25:30.526059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525936    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-model-cache" pod "2ea21d06-e50b-47f7-9921-fe38f1c010d9" (UID: "2ea21d06-e50b-47f7-9921-fe38f1c010d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:30.526059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.525958    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-home" pod "2ea21d06-e50b-47f7-9921-fe38f1c010d9" (UID: "2ea21d06-e50b-47f7-9921-fe38f1c010d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:30.526059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.526013    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tmp-dir" pod "2ea21d06-e50b-47f7-9921-fe38f1c010d9" (UID: "2ea21d06-e50b-47f7-9921-fe38f1c010d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:30.526635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.526616    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.526740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.526637    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.526740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.526648    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.527934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.527912    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tls-certs" pod "2ea21d06-e50b-47f7-9921-fe38f1c010d9" (UID: "2ea21d06-e50b-47f7-9921-fe38f1c010d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:30.528035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.528017    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea21d06-e50b-47f7-9921-fe38f1c010d9-kube-api-access-qfr9j" pod "2ea21d06-e50b-47f7-9921-fe38f1c010d9" (UID: "2ea21d06-e50b-47f7-9921-fe38f1c010d9"). InnerVolumeSpecName "kube-api-access-qfr9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:30.528325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.528307    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-dshm" pod "2ea21d06-e50b-47f7-9921-fe38f1c010d9" (UID: "2ea21d06-e50b-47f7-9921-fe38f1c010d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:30.568613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.568581    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" podUID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" containerName="payload-processing" containerID="cri-o://31df442ff7e6f3b98ff956bee23e7596443ca2ffba45e157784bfd876e8290ad" gracePeriod=25
Nov 03 20:25:30.628583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.628550    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ea21d06-e50b-47f7-9921-fe38f1c010d9-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.628583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.628583    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfr9j\" (UniqueName: \"kubernetes.io/projected/2ea21d06-e50b-47f7-9921-fe38f1c010d9-kube-api-access-qfr9j\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.628975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.628595    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ea21d06-e50b-47f7-9921-fe38f1c010d9-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:30.653061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:30.653025    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:31.338821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:31.338787    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"
Nov 03 20:25:31.338821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:31.338809    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p" event={"ID":"2ea21d06-e50b-47f7-9921-fe38f1c010d9","Type":"ContainerDied","Data":"8c0cc774a36af7a3b27a77b02561bc105ce0420978f789b2414a2c503f5599a9"}
Nov 03 20:25:31.339068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:31.338862    5282 scope.go:122] "RemoveContainer" containerID="9954b669d010d457ee5f0b91791039ce8ec5e8032e2e71e3192357f31ed50091"
Nov 03 20:25:31.363808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:31.363750    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"]
Nov 03 20:25:31.367522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:31.367497    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0a7343/test-model-0a7343-kserve-9df449995-c6d7p"]
Nov 03 20:25:31.446106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:31.446070    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:31.475009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:31.474345    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea21d06-e50b-47f7-9921-fe38f1c010d9" path="/var/lib/kubelet/pods/2ea21d06-e50b-47f7-9921-fe38f1c010d9/volumes"
Nov 03 20:25:31.617089 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:31.617012    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:25:32.299980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:32.299948    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:32.970672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:32.970632    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:33.233328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:33.233289    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:34.197415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:34.197381    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:34.985098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:34.985065    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:35.785253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:35.785178    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:35.924949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:35.924918    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:36.033938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:36.033903    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:36.223698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:36.223616    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"]
Nov 03 20:25:36.223901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:36.223875    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy" containerID="cri-o://8b941d55abe35057078c71f99a74fbc915cdc240d61079ade03fc40f2c4fad61" gracePeriod=30
Nov 03 20:25:36.227314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:36.227288    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"]
Nov 03 20:25:36.227526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:36.227491    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy" containerID="cri-o://cb2ed9455eb85fb224b7c77c4497ff3239552f8d03ab02b8a012f53958a116d9" gracePeriod=30
Nov 03 20:25:36.400752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:36.395707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:36.617693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:36.616863    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:37.340900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:37.340865    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:38.170720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:38.170657    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:38.527391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:38.527356    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" podUID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:25:38.893124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:38.893027    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:39.587895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:39.587856    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:40.323027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:40.322932    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:41.027080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.027045    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:41.176228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.175703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:41.232971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.222333    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:25:41.551980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.551950    5282 generic.go:356] "Generic (PLEG): container finished" podID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerID="cb2ed9455eb85fb224b7c77c4497ff3239552f8d03ab02b8a012f53958a116d9" exitCode=0
Nov 03 20:25:41.552211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.552050    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" event={"ID":"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2","Type":"ContainerDied","Data":"cb2ed9455eb85fb224b7c77c4497ff3239552f8d03ab02b8a012f53958a116d9"}
Nov 03 20:25:41.554211 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.554173    5282 generic.go:356] "Generic (PLEG): container finished" podID="f47726d4-a756-464a-9fff-465b9ab355b0" containerID="8b941d55abe35057078c71f99a74fbc915cdc240d61079ade03fc40f2c4fad61" exitCode=0
Nov 03 20:25:41.554345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.554331    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" event={"ID":"f47726d4-a756-464a-9fff-465b9ab355b0","Type":"ContainerDied","Data":"8b941d55abe35057078c71f99a74fbc915cdc240d61079ade03fc40f2c4fad61"}
Nov 03 20:25:41.554428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.554379    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4" event={"ID":"f47726d4-a756-464a-9fff-465b9ab355b0","Type":"ContainerDied","Data":"c1cb45b8c8d0f89cb9fe1c6e622d2eee055e77c4a6da7eadc29e987f757c7d71"}
Nov 03 20:25:41.554428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.554406    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1cb45b8c8d0f89cb9fe1c6e622d2eee055e77c4a6da7eadc29e987f757c7d71"
Nov 03 20:25:41.554428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.554425    5282 scope.go:122] "RemoveContainer" containerID="9249a1cfd31aa77756189e5fe6182125ac13a461020fb11578ed6bb72ffc2a5b"
Nov 03 20:25:41.598768 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.598661    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.627588    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-data\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.627677    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-socket\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.627708    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47726d4-a756-464a-9fff-465b9ab355b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47726d4-a756-464a-9fff-465b9ab355b0-istiod-ca-cert\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.627748    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f47726d4-a756-464a-9fff-465b9ab355b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47726d4-a756-464a-9fff-465b9ab355b0-istio-podinfo\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.627827    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-credential-socket\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.627881    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-kube-api-access-gfbjj\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-kube-api-access-gfbjj\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.627959    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-istio-token\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-istio-token\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.628066    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-envoy\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.628123    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-certs\") pod \"f47726d4-a756-464a-9fff-465b9ab355b0\" (UID: \"f47726d4-a756-464a-9fff-465b9ab355b0\") "
Nov 03 20:25:41.629800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.629447    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-certs" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.629800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.629644    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-data" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.629923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.629827    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-socket" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.630102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.630078    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47726d4-a756-464a-9fff-465b9ab355b0-istiod-ca-cert" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:41.630271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.630244    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-credential-socket" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.632465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.632438    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f47726d4-a756-464a-9fff-465b9ab355b0-istio-podinfo" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:41.633317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.633282    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-istio-token" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:41.633515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.633495    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-envoy" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.633657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.633634    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-kube-api-access-gfbjj" pod "f47726d4-a756-464a-9fff-465b9ab355b0" (UID: "f47726d4-a756-464a-9fff-465b9ab355b0"). InnerVolumeSpecName "kube-api-access-gfbjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:41.644746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.644719    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:25:41.729397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729359    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-credential-socket\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.729397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729399    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-certs\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.729609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729450    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-envoy\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.729609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729590    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-token\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-token\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.729701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729674    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-credential-socket" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.729745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729700    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-certs" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.729745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729706    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-podinfo\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.729829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729772    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-kube-api-access-nm2fw\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-kube-api-access-nm2fw\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.729829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729811    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-data\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.729929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729904    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-socket\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.730015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.729939    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istiod-ca-cert\") pod \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\" (UID: \"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2\") "
Nov 03 20:25:41.730138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730112    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-data" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.730375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730323    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-socket" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.730525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730505    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istiod-ca-cert" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:41.730820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730802    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.730908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730824    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.730908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730839    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.730908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730854    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47726d4-a756-464a-9fff-465b9ab355b0-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.730908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730868    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47726d4-a756-464a-9fff-465b9ab355b0-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.730908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730882    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.730908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730896    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730912    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730927    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfbjj\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-kube-api-access-gfbjj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730942    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730956    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730969    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47726d4-a756-464a-9fff-465b9ab355b0-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730982    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.730995    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47726d4-a756-464a-9fff-465b9ab355b0-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.731885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.731859    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-podinfo" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:41.731885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.731862    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-kube-api-access-nm2fw" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "kube-api-access-nm2fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:41.732137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.732117    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-envoy" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:41.732333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.732313    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-token" pod "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" (UID: "17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:41.831970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.831888    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.831970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.831917    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm2fw\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-kube-api-access-nm2fw\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.831970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.831928    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:41.831970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:41.831936    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:42.059351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.059315    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:42.623282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.618275    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc" event={"ID":"17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2","Type":"ContainerDied","Data":"7713a2ea543a81b019d2462f4fcaacd795485208fae91f2ecd6d61768860f13e"}
Nov 03 20:25:42.623282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.618348    5282 scope.go:122] "RemoveContainer" containerID="cb2ed9455eb85fb224b7c77c4497ff3239552f8d03ab02b8a012f53958a116d9"
Nov 03 20:25:42.623282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.618463    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"
Nov 03 20:25:42.643879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.643851    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"
Nov 03 20:25:42.854801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.854765    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"]
Nov 03 20:25:42.884290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.883012    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-qkzbc"]
Nov 03 20:25:42.893219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.890279    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"]
Nov 03 20:25:42.893219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:42.891894    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-46c34c-openshift-default-5cc965c4d9-j66w4"]
Nov 03 20:25:43.457527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:43.457490    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" path="/var/lib/kubelet/pods/17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2/volumes"
Nov 03 20:25:43.458176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:43.458154    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" path="/var/lib/kubelet/pods/f47726d4-a756-464a-9fff-465b9ab355b0/volumes"
Nov 03 20:25:43.512070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:43.512032    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:43.524382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:43.524350    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"]
Nov 03 20:25:43.595547 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:43.595492    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"]
Nov 03 20:25:43.665259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:43.665074    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:43.782243 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:43.782191    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:44.486418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:44.486387    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:44.980558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:44.980527    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"]
Nov 03 20:25:44.980772 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:44.980748    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7" podUID="20a4a27d-6002-43e8-bfa5-af8a681cbf5d" containerName="maas-api" containerID="cri-o://8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577" gracePeriod=30
Nov 03 20:25:45.056708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:45.056673    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:45.225221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:45.224868    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:45.351274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:45.350619    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:45.535181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:45.535144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:45.550442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:45.550406    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"]
Nov 03 20:25:45.711275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:45.711062    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:45.831218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:45.830952    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:46.286545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.286522    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:25:46.342629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.342589    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-openshift-service-ca\") pod \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") "
Nov 03 20:25:46.342811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.342685    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-metrics-certs\") pod \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") "
Nov 03 20:25:46.342934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.342911    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-openshift-service-ca" pod "20a4a27d-6002-43e8-bfa5-af8a681cbf5d" (UID: "20a4a27d-6002-43e8-bfa5-af8a681cbf5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:46.342988 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.342912    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-kube-api-access-h5rlg\" (UniqueName: \"kubernetes.io/projected/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-kube-api-access-h5rlg\") pod \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") "
Nov 03 20:25:46.343090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.343076    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-maas-api-tls\") pod \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\" (UID: \"20a4a27d-6002-43e8-bfa5-af8a681cbf5d\") "
Nov 03 20:25:46.343693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.343663    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:46.344897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.344873    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-metrics-certs" pod "20a4a27d-6002-43e8-bfa5-af8a681cbf5d" (UID: "20a4a27d-6002-43e8-bfa5-af8a681cbf5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:46.345323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.345297    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-kube-api-access-h5rlg" pod "20a4a27d-6002-43e8-bfa5-af8a681cbf5d" (UID: "20a4a27d-6002-43e8-bfa5-af8a681cbf5d"). InnerVolumeSpecName "kube-api-access-h5rlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:46.345502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.345484    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-maas-api-tls" pod "20a4a27d-6002-43e8-bfa5-af8a681cbf5d" (UID: "20a4a27d-6002-43e8-bfa5-af8a681cbf5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:46.445645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.445611    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5rlg\" (UniqueName: \"kubernetes.io/projected/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-kube-api-access-h5rlg\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:46.445645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.445642    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:46.445645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.445654    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20a4a27d-6002-43e8-bfa5-af8a681cbf5d-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:46.606957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.606916    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:46.791880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.791841    5282 generic.go:356] "Generic (PLEG): container finished" podID="20a4a27d-6002-43e8-bfa5-af8a681cbf5d" containerID="8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577" exitCode=0
Nov 03 20:25:46.792066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.791890    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"
Nov 03 20:25:46.792066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.791904    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7" event={"ID":"20a4a27d-6002-43e8-bfa5-af8a681cbf5d","Type":"ContainerDied","Data":"8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577"}
Nov 03 20:25:46.792066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.791958    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7" event={"ID":"20a4a27d-6002-43e8-bfa5-af8a681cbf5d","Type":"ContainerDied","Data":"22a0f52e76dd4031cbddacee98bcc568a9844830fb4ea697e73cedcd9bca4581"}
Nov 03 20:25:46.792066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.791990    5282 scope.go:122] "RemoveContainer" containerID="8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577"
Nov 03 20:25:46.803594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.803574    5282 scope.go:122] "RemoveContainer" containerID="8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577"
Nov 03 20:25:46.803889 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:46.803865    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577\": container with ID starting with 8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577 not found: ID does not exist" containerID="8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577"
Nov 03 20:25:46.803959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.803896    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577"} err="failed to get container status \"8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577\": rpc error: code = NotFound desc = could not find container \"8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577\": container with ID starting with 8d6a1d4e8ab8b8415c1ddc0b0b3feb39fdfc5f8cc0c5043625d90b2fa14db577 not found: ID does not exist"
Nov 03 20:25:46.816374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.816349    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"]
Nov 03 20:25:46.826388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:46.823734    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ee2323-f6c98f75b-47jg7"]
Nov 03 20:25:47.022388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:47.022354    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"]
Nov 03 20:25:47.110179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:47.110105    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:47.228843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:47.228797    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:47.534087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:47.533997    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a4a27d-6002-43e8-bfa5-af8a681cbf5d" path="/var/lib/kubelet/pods/20a4a27d-6002-43e8-bfa5-af8a681cbf5d/volumes"
Nov 03 20:25:47.638048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:47.638003    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:48.308377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.308332    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:48.524770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.524735    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt" podUID="321af73b-d2a5-4e37-9ee6-a6f75d0b263c" containerName="payload-processing" containerID="cri-o://cbc9b75df839dee6b5cbccb8bccaec06142139895082b651e9b6449cabeab38f" gracePeriod=25
Nov 03 20:25:48.526106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.526077    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" podUID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:25:48.596016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.595941    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2" podUID="b7156169-be29-4e7b-b323-6f18e4e09ba9" containerName="payload-pre-processing" containerID="cri-o://76c169b9736b517d19e7cfc9965a3ce8d2d30f8b7884c86e423c53ba7b6d1481" gracePeriod=25
Nov 03 20:25:48.807836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.807795    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"]
Nov 03 20:25:48.808451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.808024    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj" podUID="fe48dd6d-6155-474b-9cbd-01bc9898b66f" containerName="maas-api" containerID="cri-o://d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb" gracePeriod=30
Nov 03 20:25:48.820279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.820048    5282 generic.go:356] "Generic (PLEG): container finished" podID="b7156169-be29-4e7b-b323-6f18e4e09ba9" containerID="76c169b9736b517d19e7cfc9965a3ce8d2d30f8b7884c86e423c53ba7b6d1481" exitCode=0
Nov 03 20:25:48.821269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.821214    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2" event={"ID":"b7156169-be29-4e7b-b323-6f18e4e09ba9","Type":"ContainerDied","Data":"76c169b9736b517d19e7cfc9965a3ce8d2d30f8b7884c86e423c53ba7b6d1481"}
Nov 03 20:25:48.824169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.824145    5282 generic.go:356] "Generic (PLEG): container finished" podID="321af73b-d2a5-4e37-9ee6-a6f75d0b263c" containerID="cbc9b75df839dee6b5cbccb8bccaec06142139895082b651e9b6449cabeab38f" exitCode=0
Nov 03 20:25:48.824368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.824221    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt" event={"ID":"321af73b-d2a5-4e37-9ee6-a6f75d0b263c","Type":"ContainerDied","Data":"cbc9b75df839dee6b5cbccb8bccaec06142139895082b651e9b6449cabeab38f"}
Nov 03 20:25:48.894297 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.894257    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:48.956059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.956025    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:25:48.961618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:48.960051    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:25:49.046245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.037721    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7156169-be29-4e7b-b323-6f18e4e09ba9-kube-api-access-lw2rc\" (UniqueName: \"kubernetes.io/projected/b7156169-be29-4e7b-b323-6f18e4e09ba9-kube-api-access-lw2rc\") pod \"b7156169-be29-4e7b-b323-6f18e4e09ba9\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") "
Nov 03 20:25:49.049219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.049129    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:49.055093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.037998    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7156169-be29-4e7b-b323-6f18e4e09ba9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7156169-be29-4e7b-b323-6f18e4e09ba9-plugins-config-volume\") pod \"b7156169-be29-4e7b-b323-6f18e4e09ba9\" (UID: \"b7156169-be29-4e7b-b323-6f18e4e09ba9\") "
Nov 03 20:25:49.055093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.052812    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-plugins-config-volume\") pod \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") "
Nov 03 20:25:49.055093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.052885    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-kube-api-access-fv5xz\" (UniqueName: \"kubernetes.io/projected/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-kube-api-access-fv5xz\") pod \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\" (UID: \"321af73b-d2a5-4e37-9ee6-a6f75d0b263c\") "
Nov 03 20:25:49.058746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.058516    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-kube-api-access-fv5xz" pod "321af73b-d2a5-4e37-9ee6-a6f75d0b263c" (UID: "321af73b-d2a5-4e37-9ee6-a6f75d0b263c"). InnerVolumeSpecName "kube-api-access-fv5xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:49.059220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.059061    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-plugins-config-volume" pod "321af73b-d2a5-4e37-9ee6-a6f75d0b263c" (UID: "321af73b-d2a5-4e37-9ee6-a6f75d0b263c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:49.063135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.063108    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7156169-be29-4e7b-b323-6f18e4e09ba9-plugins-config-volume" pod "b7156169-be29-4e7b-b323-6f18e4e09ba9" (UID: "b7156169-be29-4e7b-b323-6f18e4e09ba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:49.073252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.070480    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7156169-be29-4e7b-b323-6f18e4e09ba9-kube-api-access-lw2rc" pod "b7156169-be29-4e7b-b323-6f18e4e09ba9" (UID: "b7156169-be29-4e7b-b323-6f18e4e09ba9"). InnerVolumeSpecName "kube-api-access-lw2rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:49.187071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.186985    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7156169-be29-4e7b-b323-6f18e4e09ba9-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.187326 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.187303    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.187469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.187451    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv5xz\" (UniqueName: \"kubernetes.io/projected/321af73b-d2a5-4e37-9ee6-a6f75d0b263c-kube-api-access-fv5xz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.187580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.187565    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw2rc\" (UniqueName: \"kubernetes.io/projected/b7156169-be29-4e7b-b323-6f18e4e09ba9-kube-api-access-lw2rc\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.188111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.188083    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:49.355883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.355853    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:49.365254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.365231    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:25:49.499026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.498989    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-metrics-certs\") pod \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") "
Nov 03 20:25:49.499184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.499099    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe48dd6d-6155-474b-9cbd-01bc9898b66f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe48dd6d-6155-474b-9cbd-01bc9898b66f-openshift-service-ca\") pod \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") "
Nov 03 20:25:49.499184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.499142    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-maas-api-tls\") pod \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") "
Nov 03 20:25:49.499285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.499187    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe48dd6d-6155-474b-9cbd-01bc9898b66f-kube-api-access-cjth2\" (UniqueName: \"kubernetes.io/projected/fe48dd6d-6155-474b-9cbd-01bc9898b66f-kube-api-access-cjth2\") pod \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\" (UID: \"fe48dd6d-6155-474b-9cbd-01bc9898b66f\") "
Nov 03 20:25:49.501245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.501217    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe48dd6d-6155-474b-9cbd-01bc9898b66f-openshift-service-ca" pod "fe48dd6d-6155-474b-9cbd-01bc9898b66f" (UID: "fe48dd6d-6155-474b-9cbd-01bc9898b66f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:49.503619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.503590    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe48dd6d-6155-474b-9cbd-01bc9898b66f-kube-api-access-cjth2" pod "fe48dd6d-6155-474b-9cbd-01bc9898b66f" (UID: "fe48dd6d-6155-474b-9cbd-01bc9898b66f"). InnerVolumeSpecName "kube-api-access-cjth2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:49.503740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.503594    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-maas-api-tls" pod "fe48dd6d-6155-474b-9cbd-01bc9898b66f" (UID: "fe48dd6d-6155-474b-9cbd-01bc9898b66f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:49.504340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.504312    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-metrics-certs" pod "fe48dd6d-6155-474b-9cbd-01bc9898b66f" (UID: "fe48dd6d-6155-474b-9cbd-01bc9898b66f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:49.601629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.601597    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.601629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.601626    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjth2\" (UniqueName: \"kubernetes.io/projected/fe48dd6d-6155-474b-9cbd-01bc9898b66f-kube-api-access-cjth2\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.601837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.601637    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe48dd6d-6155-474b-9cbd-01bc9898b66f-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.601837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.601646    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe48dd6d-6155-474b-9cbd-01bc9898b66f-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:49.706378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.706328    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:49.833498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.833417    5282 generic.go:356] "Generic (PLEG): container finished" podID="fe48dd6d-6155-474b-9cbd-01bc9898b66f" containerID="d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb" exitCode=0
Nov 03 20:25:49.833498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.833458    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"
Nov 03 20:25:49.833953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.833495    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj" event={"ID":"fe48dd6d-6155-474b-9cbd-01bc9898b66f","Type":"ContainerDied","Data":"d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb"}
Nov 03 20:25:49.833953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.833544    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj" event={"ID":"fe48dd6d-6155-474b-9cbd-01bc9898b66f","Type":"ContainerDied","Data":"bbfca0a5706ea9f809d5b0989826a67f9cf582d4d1cccf588e3bbfa1d990d7b6"}
Nov 03 20:25:49.833953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.833569    5282 scope.go:122] "RemoveContainer" containerID="d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb"
Nov 03 20:25:49.835264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.835237    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt" event={"ID":"321af73b-d2a5-4e37-9ee6-a6f75d0b263c","Type":"ContainerDied","Data":"d3c4f14b2a7c2755449cfb170ee08686144ee064917290b62c64f1e9dd40e742"}
Nov 03 20:25:49.835264 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.835259    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"
Nov 03 20:25:49.836888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.836864    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2" event={"ID":"b7156169-be29-4e7b-b323-6f18e4e09ba9","Type":"ContainerDied","Data":"762b28008966e4072be9f568a80b6a7facbb326e25868a36293d060f837b3d13"}
Nov 03 20:25:49.836888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.836876    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"
Nov 03 20:25:49.846181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.846166    5282 scope.go:122] "RemoveContainer" containerID="d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb"
Nov 03 20:25:49.846451 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:49.846433    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb\": container with ID starting with d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb not found: ID does not exist" containerID="d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb"
Nov 03 20:25:49.846509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.846460    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb"} err="failed to get container status \"d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb\": rpc error: code = NotFound desc = could not find container \"d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb\": container with ID starting with d983c3147a69959c4d106d50c939876cae41ef73cdc55e1139a672a8e4a06bcb not found: ID does not exist"
Nov 03 20:25:49.846509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.846477    5282 scope.go:122] "RemoveContainer" containerID="cbc9b75df839dee6b5cbccb8bccaec06142139895082b651e9b6449cabeab38f"
Nov 03 20:25:49.855994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.855972    5282 scope.go:122] "RemoveContainer" containerID="76c169b9736b517d19e7cfc9965a3ce8d2d30f8b7884c86e423c53ba7b6d1481"
Nov 03 20:25:49.862478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.862453    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"]
Nov 03 20:25:49.866479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.866456    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-ee2323-75ffb95c7f-sl6s2"]
Nov 03 20:25:49.875844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.875822    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"]
Nov 03 20:25:49.878849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.878829    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-ee2323-7889c4b54d-g98tt"]
Nov 03 20:25:49.891707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.891684    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"]
Nov 03 20:25:49.898092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:49.898063    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a7343-57d64cf499-prttj"]
Nov 03 20:25:50.193766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.193683    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:50.850508 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.850477    5282 generic.go:356] "Generic (PLEG): container finished" podID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" containerID="31df442ff7e6f3b98ff956bee23e7596443ca2ffba45e157784bfd876e8290ad" exitCode=0
Nov 03 20:25:50.850823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.850512    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" event={"ID":"28ca8d9b-bda8-46c9-bb3e-69aa77e06445","Type":"ContainerDied","Data":"31df442ff7e6f3b98ff956bee23e7596443ca2ffba45e157784bfd876e8290ad"}
Nov 03 20:25:50.850823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.850571    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j" event={"ID":"28ca8d9b-bda8-46c9-bb3e-69aa77e06445","Type":"ContainerDied","Data":"526f90cace1d3dbb6c2995d16336e68c9379d692267a89778c2d6843ccb62125"}
Nov 03 20:25:50.850823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.850587    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="526f90cace1d3dbb6c2995d16336e68c9379d692267a89778c2d6843ccb62125"
Nov 03 20:25:50.862804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.862781    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:25:50.938874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.938829    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-plugins-config-volume\") pod \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") "
Nov 03 20:25:50.939080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.938929    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-kube-api-access-dr8lj\" (UniqueName: \"kubernetes.io/projected/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-kube-api-access-dr8lj\") pod \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\" (UID: \"28ca8d9b-bda8-46c9-bb3e-69aa77e06445\") "
Nov 03 20:25:50.939342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.939307    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-plugins-config-volume" pod "28ca8d9b-bda8-46c9-bb3e-69aa77e06445" (UID: "28ca8d9b-bda8-46c9-bb3e-69aa77e06445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:50.939924 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.939896    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:50.946239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:50.942116    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-kube-api-access-dr8lj" pod "28ca8d9b-bda8-46c9-bb3e-69aa77e06445" (UID: "28ca8d9b-bda8-46c9-bb3e-69aa77e06445"). InnerVolumeSpecName "kube-api-access-dr8lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:51.042236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.042191    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr8lj\" (UniqueName: \"kubernetes.io/projected/28ca8d9b-bda8-46c9-bb3e-69aa77e06445-kube-api-access-dr8lj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:51.130301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.130183    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:51.457949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.457872    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321af73b-d2a5-4e37-9ee6-a6f75d0b263c" path="/var/lib/kubelet/pods/321af73b-d2a5-4e37-9ee6-a6f75d0b263c/volumes"
Nov 03 20:25:51.458390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.458375    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7156169-be29-4e7b-b323-6f18e4e09ba9" path="/var/lib/kubelet/pods/b7156169-be29-4e7b-b323-6f18e4e09ba9/volumes"
Nov 03 20:25:51.458836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.458821    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe48dd6d-6155-474b-9cbd-01bc9898b66f" path="/var/lib/kubelet/pods/fe48dd6d-6155-474b-9cbd-01bc9898b66f/volumes"
Nov 03 20:25:51.612356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.612319    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:51.719273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.719157    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:51.877248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.870581    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"
Nov 03 20:25:51.963235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.962242    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:51.963235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.962314    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"]
Nov 03 20:25:51.975273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:51.972873    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-46c34c-6bcdcf5f5d-t8p8j"]
Nov 03 20:25:52.033155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.026051    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" podUID="ca057d53-559b-4e56-9635-24ef10b0b325" containerName="payload-pre-processing" containerID="cri-o://f80cdbc0aadd09a85a75dba987b8865421f4f6dc5e36f28fdab7bc8d4c3b8133" gracePeriod=25
Nov 03 20:25:52.085907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.085876    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:52.227964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.227640    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:52.246191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.246153    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"]
Nov 03 20:25:52.246462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.246432    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" containerID="cri-o://aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d" gracePeriod=30
Nov 03 20:25:52.252775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.252748    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"]
Nov 03 20:25:52.252955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.252931    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy" containerID="cri-o://57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b" gracePeriod=30
Nov 03 20:25:52.258177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.258153    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"]
Nov 03 20:25:52.258417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.258388    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy" containerID="cri-o://f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac" gracePeriod=30
Nov 03 20:25:52.392236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:52.391428    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:53.099083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:53.099036    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:53.462995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:53.462898    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" path="/var/lib/kubelet/pods/28ca8d9b-bda8-46c9-bb3e-69aa77e06445/volumes"
Nov 03 20:25:53.807619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:53.807579    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:54.511106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:54.511070    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:55.108119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:55.108085    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:55.203068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:55.203028    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"]
Nov 03 20:25:55.203308 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:55.203276    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy" containerID="cri-o://dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20" gracePeriod=30
Nov 03 20:25:55.869220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:55.863723    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.097738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.097705    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.225207 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.225108    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.354866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.354812    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"]
Nov 03 20:25:56.355119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.355089    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy" containerID="cri-o://8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7" gracePeriod=30
Nov 03 20:25:56.597220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.573554    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.678814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.678778    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:25:56.679108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.679077    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:25:56.788606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.788572    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:56.959272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:56.952001    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:57.345448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.344761    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" podUID="ca057d53-559b-4e56-9635-24ef10b0b325" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:25:57.580352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.567936    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:25:57.607254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.604455    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:57.619256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.619227    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:25:57.623124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.623099    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:25:57.627270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627157    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-envoy\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.627270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627225    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-certs\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.627466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627271    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-kube-api-access-s2hfl\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-kube-api-access-s2hfl\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.627466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627302    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-certs\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.627466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627334    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-data\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.627466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627373    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-token\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-token\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.627466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627437    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-credential-socket\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.627735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627525    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-data\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.627735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627554    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-kube-api-access-zt6cx\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-kube-api-access-zt6cx\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.627735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627630    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-envoy\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.627735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627676    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-certs\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.627735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627704    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-credential-socket\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.627996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627755    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-socket\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.627996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627789    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-socket\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.627996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627846    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istiod-ca-cert\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.627996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.627899    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-socket\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.628260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628062    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-istio-token\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-istio-token\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.628260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628103    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f42331-cba4-451d-b86c-89dc9e983332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18f42331-cba4-451d-b86c-89dc9e983332-istiod-ca-cert\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.628260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628138    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-kube-api-access-thbwf\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-kube-api-access-thbwf\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.628260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628186    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istiod-ca-cert\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.628510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628295    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-podinfo\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.628510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628397    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-credential-socket\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.628510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628429    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-data\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.628510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628466    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-podinfo\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.628716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628515    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-envoy\") pod \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\" (UID: \"690ea14a-8ce1-4776-a478-fd9a2b66b4c9\") "
Nov 03 20:25:57.628716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628570    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18f42331-cba4-451d-b86c-89dc9e983332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18f42331-cba4-451d-b86c-89dc9e983332-istio-podinfo\") pod \"18f42331-cba4-451d-b86c-89dc9e983332\" (UID: \"18f42331-cba4-451d-b86c-89dc9e983332\") "
Nov 03 20:25:57.629062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.628961    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f42331-cba4-451d-b86c-89dc9e983332-istiod-ca-cert" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:57.629111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.629095    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-token\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-token\") pod \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\" (UID: \"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8\") "
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.629482    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-credential-socket" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.629884    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-socket" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630056    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-data" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630115    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-certs" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630127    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-certs" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630254    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-socket" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630377    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-credential-socket" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630451    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-credential-socket" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630599    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-socket" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630616    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istiod-ca-cert" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:57.630671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.630651    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-certs" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.631753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.631051    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-data" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.631753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.631406    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istiod-ca-cert" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:57.631753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.631594    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-envoy" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.631971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.631950    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.631978    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.631993    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632007    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632021    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632034    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632048    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632067    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632081    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632095    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632110    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632124    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632138    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18f42331-cba4-451d-b86c-89dc9e983332-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632152    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.632283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632167    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.633389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632641    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-token" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:57.633389 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.632970    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-data" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.633609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.633590    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-kube-api-access-zt6cx" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "kube-api-access-zt6cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:57.638366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.638341    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-podinfo" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:57.638915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.638871    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-envoy" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.639556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.639376    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-kube-api-access-thbwf" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "kube-api-access-thbwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:57.639556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.639426    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-podinfo" pod "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" (UID: "c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:57.639556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.639424    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-token" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:57.639556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.639504    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-istio-token" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:57.640725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.640693    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18f42331-cba4-451d-b86c-89dc9e983332-istio-podinfo" pod "18f42331-cba4-451d-b86c-89dc9e983332" (UID: "18f42331-cba4-451d-b86c-89dc9e983332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:57.640929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.640902    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-kube-api-access-s2hfl" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "kube-api-access-s2hfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:57.642409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.642379    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-envoy" pod "690ea14a-8ce1-4776-a478-fd9a2b66b4c9" (UID: "690ea14a-8ce1-4776-a478-fd9a2b66b4c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:57.733531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733493    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733526    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thbwf\" (UniqueName: \"kubernetes.io/projected/18f42331-cba4-451d-b86c-89dc9e983332-kube-api-access-thbwf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733537    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733547    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18f42331-cba4-451d-b86c-89dc9e983332-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733556    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733565    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733574    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18f42331-cba4-451d-b86c-89dc9e983332-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733582    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733590    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2hfl\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-kube-api-access-s2hfl\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733598    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/690ea14a-8ce1-4776-a478-fd9a2b66b4c9-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733606    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt6cx\" (UniqueName: \"kubernetes.io/projected/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-kube-api-access-zt6cx\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.733773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.733615    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:57.864280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:57.855593    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:58.015036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.009297    5282 generic.go:356] "Generic (PLEG): container finished" podID="18f42331-cba4-451d-b86c-89dc9e983332" containerID="f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac" exitCode=0
Nov 03 20:25:58.015036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.009546    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" event={"ID":"18f42331-cba4-451d-b86c-89dc9e983332","Type":"ContainerDied","Data":"f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac"}
Nov 03 20:25:58.015036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.009637    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5" event={"ID":"18f42331-cba4-451d-b86c-89dc9e983332","Type":"ContainerDied","Data":"b2fd137bd6728a9e647f4cb863a026da8cf0b982f356b9bb999470a838ed5ac0"}
Nov 03 20:25:58.015036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.009670    5282 scope.go:122] "RemoveContainer" containerID="f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac"
Nov 03 20:25:58.015036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.009865    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"
Nov 03 20:25:58.028533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.028488    5282 generic.go:356] "Generic (PLEG): container finished" podID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerID="57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b" exitCode=0
Nov 03 20:25:58.028890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.028854    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" event={"ID":"690ea14a-8ce1-4776-a478-fd9a2b66b4c9","Type":"ContainerDied","Data":"57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b"}
Nov 03 20:25:58.029060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.029045    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s" event={"ID":"690ea14a-8ce1-4776-a478-fd9a2b66b4c9","Type":"ContainerDied","Data":"abc3a986561bd73cde0a7c2acea76d188f683fbdb926b379ec9f21e431d1c91a"}
Nov 03 20:25:58.029296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.029270    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"
Nov 03 20:25:58.048402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.047484    5282 generic.go:356] "Generic (PLEG): container finished" podID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerID="aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d" exitCode=0
Nov 03 20:25:58.048402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.047556    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" event={"ID":"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8","Type":"ContainerDied","Data":"aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d"}
Nov 03 20:25:58.048402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.047608    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk" event={"ID":"c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8","Type":"ContainerDied","Data":"7fedd3c84532e6e87fbe96396a10743ea8fcdb753e8e80287b3f6f1b15869e6c"}
Nov 03 20:25:58.048402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.047716    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"
Nov 03 20:25:58.073954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.073925    5282 scope.go:122] "RemoveContainer" containerID="f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac"
Nov 03 20:25:58.074324 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:58.074293    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac\": container with ID starting with f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac not found: ID does not exist" containerID="f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac"
Nov 03 20:25:58.074447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.074337    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac"} err="failed to get container status \"f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac\": rpc error: code = NotFound desc = could not find container \"f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac\": container with ID starting with f44362cff6aa65893029c70cef2bbf92984b008ac0d358934d2d46bb08ff80ac not found: ID does not exist"
Nov 03 20:25:58.074447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.074363    5282 scope.go:122] "RemoveContainer" containerID="57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b"
Nov 03 20:25:58.090002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.089978    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"]
Nov 03 20:25:58.099560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.099529    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-zxv2s"]
Nov 03 20:25:58.106372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.105957    5282 scope.go:122] "RemoveContainer" containerID="57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b"
Nov 03 20:25:58.106855 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:58.106588    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b\": container with ID starting with 57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b not found: ID does not exist" containerID="57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b"
Nov 03 20:25:58.106855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.106663    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b"} err="failed to get container status \"57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b\": rpc error: code = NotFound desc = could not find container \"57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b\": container with ID starting with 57e21b76c2f7f2d787ab58a3c3181a9a43c93aa85eff5a2bc572b685a936be4b not found: ID does not exist"
Nov 03 20:25:58.106855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.106690    5282 scope.go:122] "RemoveContainer" containerID="aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d"
Nov 03 20:25:58.120651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.120581    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"]
Nov 03 20:25:58.127271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.127225    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-lsjs5"]
Nov 03 20:25:58.133694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.133670    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"]
Nov 03 20:25:58.136559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.136536    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ee2323-openshift-default-855486c599-ntzdk"]
Nov 03 20:25:58.138755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.138740    5282 scope.go:122] "RemoveContainer" containerID="ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023"
Nov 03 20:25:58.150912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.150887    5282 scope.go:122] "RemoveContainer" containerID="aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d"
Nov 03 20:25:58.151274 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:58.151252    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d\": container with ID starting with aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d not found: ID does not exist" containerID="aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d"
Nov 03 20:25:58.151373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.151284    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d"} err="failed to get container status \"aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d\": rpc error: code = NotFound desc = could not find container \"aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d\": container with ID starting with aa61052911ba091574042ca9126a84e54a4263c03b953f368ecf0600e0d5a79d not found: ID does not exist"
Nov 03 20:25:58.151373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.151309    5282 scope.go:122] "RemoveContainer" containerID="ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023"
Nov 03 20:25:58.151632 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:25:58.151610    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023\": container with ID starting with ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023 not found: ID does not exist" containerID="ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023"
Nov 03 20:25:58.151709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.151637    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023"} err="failed to get container status \"ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023\": rpc error: code = NotFound desc = could not find container \"ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023\": container with ID starting with ae13294a2c78c62130f0541b541aefb2c590d841b625a8f02c807fff67660023 not found: ID does not exist"
Nov 03 20:25:58.218895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.218861    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:58.339079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.338223    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:58.660229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:58.652640    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:59.052191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:59.052138    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:25:59.456897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:59.456818    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f42331-cba4-451d-b86c-89dc9e983332" path="/var/lib/kubelet/pods/18f42331-cba4-451d-b86c-89dc9e983332/volumes"
Nov 03 20:25:59.457346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:59.457328    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" path="/var/lib/kubelet/pods/690ea14a-8ce1-4776-a478-fd9a2b66b4c9/volumes"
Nov 03 20:25:59.457796 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:59.457782    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" path="/var/lib/kubelet/pods/c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8/volumes"
Nov 03 20:25:59.765437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:25:59.765399    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:00.487589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.487567    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:26:00.526935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.526904    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-kube-api-access-lk4fr\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-kube-api-access-lk4fr\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.527132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.526949    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-credential-socket\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.527132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.526991    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-data\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.527349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527292    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-credential-socket" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.527349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527325    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-data" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.527548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527346    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-podinfo\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.527548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527539    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-envoy\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527756    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istiod-ca-cert\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527829    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-socket\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527882    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-certs\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.527956    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-token\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-token\") pod \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\" (UID: \"f008b1c7-64d5-4ed3-8954-b789905c0f0e\") "
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.528085    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istiod-ca-cert" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.528348    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-socket" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.528661    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.528683    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.528698    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.528710    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.528969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.528924    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-certs" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.529871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.529836    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-kube-api-access-lk4fr" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "kube-api-access-lk4fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:00.530776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.530742    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-podinfo" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:00.531011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.530981    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-token" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:00.537698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.537670    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-envoy" pod "f008b1c7-64d5-4ed3-8954-b789905c0f0e" (UID: "f008b1c7-64d5-4ed3-8954-b789905c0f0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.538691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.538581    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:00.586741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.586702    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" podUID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" containerName="main" containerID="cri-o://fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557" gracePeriod=45
Nov 03 20:26:00.648563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.647510    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.648563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.647554    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.648563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.647572    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4fr\" (UniqueName: \"kubernetes.io/projected/f008b1c7-64d5-4ed3-8954-b789905c0f0e-kube-api-access-lk4fr\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.648563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.647587    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.648563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.647601    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f008b1c7-64d5-4ed3-8954-b789905c0f0e-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.871840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.871810    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:26:00.951354 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951326    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-model-cache\") pod \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") "
Nov 03 20:26:00.951534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951404    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tmp-dir\") pod \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") "
Nov 03 20:26:00.951534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951493    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-dshm\") pod \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") "
Nov 03 20:26:00.951644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951605    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-home\") pod \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") "
Nov 03 20:26:00.951644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951626    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-model-cache" pod "c7d8c32f-89d6-403e-ac8f-47fcf85f2632" (UID: "c7d8c32f-89d6-403e-ac8f-47fcf85f2632"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.951780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951656    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tmp-dir" pod "c7d8c32f-89d6-403e-ac8f-47fcf85f2632" (UID: "c7d8c32f-89d6-403e-ac8f-47fcf85f2632"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.951843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951815    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-kube-api-access-b5hht\" (UniqueName: \"kubernetes.io/projected/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-kube-api-access-b5hht\") pod \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") "
Nov 03 20:26:00.951843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951833    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-home" pod "c7d8c32f-89d6-403e-ac8f-47fcf85f2632" (UID: "c7d8c32f-89d6-403e-ac8f-47fcf85f2632"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.951962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.951862    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tls-certs\") pod \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\" (UID: \"c7d8c32f-89d6-403e-ac8f-47fcf85f2632\") "
Nov 03 20:26:00.952445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.952424    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.952582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.952449    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.952582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.952461    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:00.953993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.953974    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tls-certs" pod "c7d8c32f-89d6-403e-ac8f-47fcf85f2632" (UID: "c7d8c32f-89d6-403e-ac8f-47fcf85f2632"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:00.954149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.954133    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-dshm" pod "c7d8c32f-89d6-403e-ac8f-47fcf85f2632" (UID: "c7d8c32f-89d6-403e-ac8f-47fcf85f2632"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:00.954238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:00.954146    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-kube-api-access-b5hht" pod "c7d8c32f-89d6-403e-ac8f-47fcf85f2632" (UID: "c7d8c32f-89d6-403e-ac8f-47fcf85f2632"). InnerVolumeSpecName "kube-api-access-b5hht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:01.054432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.054350    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hht\" (UniqueName: \"kubernetes.io/projected/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-kube-api-access-b5hht\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.054432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.054385    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.054432 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.054401    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8c32f-89d6-403e-ac8f-47fcf85f2632-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.075102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.075068    5282 generic.go:356] "Generic (PLEG): container finished" podID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerID="dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20" exitCode=0
Nov 03 20:26:01.075298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.075125    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"
Nov 03 20:26:01.075298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.075153    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" event={"ID":"f008b1c7-64d5-4ed3-8954-b789905c0f0e","Type":"ContainerDied","Data":"dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20"}
Nov 03 20:26:01.075298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.075228    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s" event={"ID":"f008b1c7-64d5-4ed3-8954-b789905c0f0e","Type":"ContainerDied","Data":"cf0e660d06df7761ca479188091db16d010301871ed853289cc34834bf5763a5"}
Nov 03 20:26:01.075298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.075255    5282 scope.go:122] "RemoveContainer" containerID="dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20"
Nov 03 20:26:01.076691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.076663    5282 generic.go:356] "Generic (PLEG): container finished" podID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" containerID="fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557" exitCode=0
Nov 03 20:26:01.076766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.076703    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"
Nov 03 20:26:01.076824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.076769    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" event={"ID":"c7d8c32f-89d6-403e-ac8f-47fcf85f2632","Type":"ContainerDied","Data":"fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557"}
Nov 03 20:26:01.076824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.076800    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr" event={"ID":"c7d8c32f-89d6-403e-ac8f-47fcf85f2632","Type":"ContainerDied","Data":"643592726d9f1cfd60e6731e1740d4362056548a9ba2120ec7580a884f057c66"}
Nov 03 20:26:01.086654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.086483    5282 scope.go:122] "RemoveContainer" containerID="dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20"
Nov 03 20:26:01.086884 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:01.086855    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20\": container with ID starting with dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20 not found: ID does not exist" containerID="dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20"
Nov 03 20:26:01.086966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.086890    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20"} err="failed to get container status \"dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20\": rpc error: code = NotFound desc = could not find container \"dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20\": container with ID starting with dc0813593b38e84c8a7ba6c85a6cd2bf0852f21697a2dedb8721edd43bdd9d20 not found: ID does not exist"
Nov 03 20:26:01.086966 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.086911    5282 scope.go:122] "RemoveContainer" containerID="fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557"
Nov 03 20:26:01.096415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.096398    5282 scope.go:122] "RemoveContainer" containerID="fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557"
Nov 03 20:26:01.096663 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:01.096644    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557\": container with ID starting with fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557 not found: ID does not exist" containerID="fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557"
Nov 03 20:26:01.096747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.096676    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557"} err="failed to get container status \"fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557\": rpc error: code = NotFound desc = could not find container \"fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557\": container with ID starting with fc2c4cc54996af7cc7cd6b0137c4ac4c2c5a9d62d2ad519911eb70a8f6fe1557 not found: ID does not exist"
Nov 03 20:26:01.110679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.110648    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"]
Nov 03 20:26:01.128265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.128226    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-2hv5s"]
Nov 03 20:26:01.137306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.137272    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"]
Nov 03 20:26:01.140846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.140820    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-22a567/test-model-22a567-kserve-6d86688d85-pcrpr"]
Nov 03 20:26:01.422753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.422718    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:01.502210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.502156    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" path="/var/lib/kubelet/pods/c7d8c32f-89d6-403e-ac8f-47fcf85f2632/volumes"
Nov 03 20:26:01.503002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.502982    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" path="/var/lib/kubelet/pods/f008b1c7-64d5-4ed3-8954-b789905c0f0e/volumes"
Nov 03 20:26:01.622802 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:01.622713    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:26:01.670683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.670651    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:26:01.789373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789337    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de77257b-93d9-472b-a007-a6d199acf1e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de77257b-93d9-472b-a007-a6d199acf1e2-istio-podinfo\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.789597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789404    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-istio-token\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-istio-token\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.789597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789520    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-data\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.789597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789567    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-envoy\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.789776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789613    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-credential-socket\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.789776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789745    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-certs\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.789879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789763    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-data" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:01.789937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789894    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-kube-api-access-vt6ft\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-kube-api-access-vt6ft\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.789992 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.789930    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-credential-socket" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:01.790045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790008    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-socket\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.790045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790029    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-certs" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:01.790158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790067    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de77257b-93d9-472b-a007-a6d199acf1e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de77257b-93d9-472b-a007-a6d199acf1e2-istiod-ca-cert\") pod \"de77257b-93d9-472b-a007-a6d199acf1e2\" (UID: \"de77257b-93d9-472b-a007-a6d199acf1e2\") "
Nov 03 20:26:01.790457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790394    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-socket" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:01.790584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790527    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de77257b-93d9-472b-a007-a6d199acf1e2-istiod-ca-cert" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:01.790802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790781    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.790845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790811    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de77257b-93d9-472b-a007-a6d199acf1e2-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.790845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790827    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.790845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790841    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.790949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.790855    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.791842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.791814    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-envoy" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:01.791968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.791901    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-istio-token" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:01.792072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.792055    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de77257b-93d9-472b-a007-a6d199acf1e2-istio-podinfo" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:01.792572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.792553    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-kube-api-access-vt6ft" pod "de77257b-93d9-472b-a007-a6d199acf1e2" (UID: "de77257b-93d9-472b-a007-a6d199acf1e2"). InnerVolumeSpecName "kube-api-access-vt6ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:01.892408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.892318    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de77257b-93d9-472b-a007-a6d199acf1e2-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.892408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.892353    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.892408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.892363    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de77257b-93d9-472b-a007-a6d199acf1e2-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:01.892408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:01.892372    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt6ft\" (UniqueName: \"kubernetes.io/projected/de77257b-93d9-472b-a007-a6d199acf1e2-kube-api-access-vt6ft\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:02.086898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.086866    5282 generic.go:356] "Generic (PLEG): container finished" podID="de77257b-93d9-472b-a007-a6d199acf1e2" containerID="8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7" exitCode=0
Nov 03 20:26:02.087056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.086913    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"
Nov 03 20:26:02.087056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.086951    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" event={"ID":"de77257b-93d9-472b-a007-a6d199acf1e2","Type":"ContainerDied","Data":"8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7"}
Nov 03 20:26:02.087056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.087007    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl" event={"ID":"de77257b-93d9-472b-a007-a6d199acf1e2","Type":"ContainerDied","Data":"795106f6d51744ee6e1fee8954ea0404348374fa6b11ff5a229bdfbcf9f13607"}
Nov 03 20:26:02.087056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.087036    5282 scope.go:122] "RemoveContainer" containerID="8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7"
Nov 03 20:26:02.100318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.099800    5282 scope.go:122] "RemoveContainer" containerID="6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634"
Nov 03 20:26:02.115360 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.115327    5282 scope.go:122] "RemoveContainer" containerID="8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7"
Nov 03 20:26:02.115680 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:02.115655    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7\": container with ID starting with 8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7 not found: ID does not exist" containerID="8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7"
Nov 03 20:26:02.115737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.115691    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7"} err="failed to get container status \"8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7\": rpc error: code = NotFound desc = could not find container \"8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7\": container with ID starting with 8087e8fb613ac66d7975bcb37f548d531ab55018782eba6188fc64a2128685e7 not found: ID does not exist"
Nov 03 20:26:02.115737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.115714    5282 scope.go:122] "RemoveContainer" containerID="6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634"
Nov 03 20:26:02.116010 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:02.115990    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634\": container with ID starting with 6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634 not found: ID does not exist" containerID="6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634"
Nov 03 20:26:02.116073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.116018    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634"} err="failed to get container status \"6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634\": rpc error: code = NotFound desc = could not find container \"6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634\": container with ID starting with 6c656cd4660dc35b1169214689f1884057328ba945032fc6f7a39d80a60f0634 not found: ID does not exist"
Nov 03 20:26:02.116147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.116127    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"]
Nov 03 20:26:02.124081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.124060    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a7343-openshift-default-74585db445-lplwl"]
Nov 03 20:26:02.125723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.125705    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:02.642383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.642349    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"]
Nov 03 20:26:02.642789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.642587    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy" containerID="cri-o://796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1" gracePeriod=30
Nov 03 20:26:02.858974 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.858943    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"]
Nov 03 20:26:02.956515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:02.956291    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:03.076242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:03.076166    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:03.456090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:03.456059    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" path="/var/lib/kubelet/pods/de77257b-93d9-472b-a007-a6d199acf1e2/volumes"
Nov 03 20:26:03.591755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:03.591722    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"]
Nov 03 20:26:03.592032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:03.591967    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c" podUID="0972f4db-9691-4eb9-94a2-5dd461e656dd" containerName="maas-api" containerID="cri-o://0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f" gracePeriod=30
Nov 03 20:26:03.680570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:03.680537    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:03.895220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:03.874900    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:04.015219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.009038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:04.143930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.143903    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:26:04.162521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.162489    5282 generic.go:356] "Generic (PLEG): container finished" podID="0972f4db-9691-4eb9-94a2-5dd461e656dd" containerID="0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f" exitCode=0
Nov 03 20:26:04.162682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.162528    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"
Nov 03 20:26:04.162682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.162564    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c" event={"ID":"0972f4db-9691-4eb9-94a2-5dd461e656dd","Type":"ContainerDied","Data":"0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f"}
Nov 03 20:26:04.162682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.162624    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c" event={"ID":"0972f4db-9691-4eb9-94a2-5dd461e656dd","Type":"ContainerDied","Data":"2ea4a99c8bbbc308ffcdfb3113ab3627eec6adfd17662b9dd9a9a2e5181bf4f0"}
Nov 03 20:26:04.162682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.162638    5282 scope.go:122] "RemoveContainer" containerID="0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f"
Nov 03 20:26:04.171369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.171107    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:04.173946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.173927    5282 scope.go:122] "RemoveContainer" containerID="0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f"
Nov 03 20:26:04.174267 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:04.174244    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f\": container with ID starting with 0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f not found: ID does not exist" containerID="0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f"
Nov 03 20:26:04.174367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.174273    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f"} err="failed to get container status \"0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f\": rpc error: code = NotFound desc = could not find container \"0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f\": container with ID starting with 0866ef83c008c53ae78b47733618ebd7e5f0a7fb5a8659ea1032ae23902bbf8f not found: ID does not exist"
Nov 03 20:26:04.340280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.319749    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-metrics-certs\") pod \"0972f4db-9691-4eb9-94a2-5dd461e656dd\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") "
Nov 03 20:26:04.340280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.319826    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0972f4db-9691-4eb9-94a2-5dd461e656dd-kube-api-access-wbd6m\" (UniqueName: \"kubernetes.io/projected/0972f4db-9691-4eb9-94a2-5dd461e656dd-kube-api-access-wbd6m\") pod \"0972f4db-9691-4eb9-94a2-5dd461e656dd\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") "
Nov 03 20:26:04.340280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.319898    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls\") pod \"0972f4db-9691-4eb9-94a2-5dd461e656dd\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") "
Nov 03 20:26:04.340280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.319991    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0972f4db-9691-4eb9-94a2-5dd461e656dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0972f4db-9691-4eb9-94a2-5dd461e656dd-openshift-service-ca\") pod \"0972f4db-9691-4eb9-94a2-5dd461e656dd\" (UID: \"0972f4db-9691-4eb9-94a2-5dd461e656dd\") "
Nov 03 20:26:04.340280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.321761    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:04.341240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.340874    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0972f4db-9691-4eb9-94a2-5dd461e656dd-openshift-service-ca" pod "0972f4db-9691-4eb9-94a2-5dd461e656dd" (UID: "0972f4db-9691-4eb9-94a2-5dd461e656dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:04.381444 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.376401    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0972f4db-9691-4eb9-94a2-5dd461e656dd-kube-api-access-wbd6m" pod "0972f4db-9691-4eb9-94a2-5dd461e656dd" (UID: "0972f4db-9691-4eb9-94a2-5dd461e656dd"). InnerVolumeSpecName "kube-api-access-wbd6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:04.386301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.381881    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls" pod "0972f4db-9691-4eb9-94a2-5dd461e656dd" (UID: "0972f4db-9691-4eb9-94a2-5dd461e656dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:04.402407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.402366    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-metrics-certs" pod "0972f4db-9691-4eb9-94a2-5dd461e656dd" (UID: "0972f4db-9691-4eb9-94a2-5dd461e656dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:04.425536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.425496    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:04.425763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.425742    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbd6m\" (UniqueName: \"kubernetes.io/projected/0972f4db-9691-4eb9-94a2-5dd461e656dd-kube-api-access-wbd6m\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:04.425857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.425844    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0972f4db-9691-4eb9-94a2-5dd461e656dd-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:04.425957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.425945    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0972f4db-9691-4eb9-94a2-5dd461e656dd-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:04.458929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.458893    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:04.504780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.503483    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"]
Nov 03 20:26:04.515095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.515058    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-0ffd89-5555485f68-fbd9c"]
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.779873    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"]
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781038    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" containerName="main"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781065    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" containerName="main"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781107    5282 cpu_manager.go:414] "containerMap: removing container" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781117    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781129    5282 cpu_manager.go:414] "containerMap: removing container" podUID="20a4a27d-6002-43e8-bfa5-af8a681cbf5d" containerName="maas-api"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781137    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a4a27d-6002-43e8-bfa5-af8a681cbf5d" containerName="maas-api"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781163    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781170    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781182    5282 cpu_manager.go:414] "containerMap: removing container" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781189    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781237    5282 cpu_manager.go:414] "containerMap: removing container" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781246    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781257    5282 cpu_manager.go:414] "containerMap: removing container" podUID="2ea21d06-e50b-47f7-9921-fe38f1c010d9" containerName="main"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781264    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea21d06-e50b-47f7-9921-fe38f1c010d9" containerName="main"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781284    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781291    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781300    5282 cpu_manager.go:414] "containerMap: removing container" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781308    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781329    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781337    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781355    5282 cpu_manager.go:414] "containerMap: removing container" podUID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" containerName="payload-processing"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781363    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" containerName="payload-processing"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781373    5282 cpu_manager.go:414] "containerMap: removing container" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781380    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781392    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b272424b-a840-4c8f-9622-09136854de34" containerName="main"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781400    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b272424b-a840-4c8f-9622-09136854de34" containerName="main"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781420    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0972f4db-9691-4eb9-94a2-5dd461e656dd" containerName="maas-api"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781429    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0972f4db-9691-4eb9-94a2-5dd461e656dd" containerName="maas-api"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781443    5282 cpu_manager.go:414] "containerMap: removing container" podUID="fe48dd6d-6155-474b-9cbd-01bc9898b66f" containerName="maas-api"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781450    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe48dd6d-6155-474b-9cbd-01bc9898b66f" containerName="maas-api"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781652    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b7156169-be29-4e7b-b323-6f18e4e09ba9" containerName="payload-pre-processing"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781661    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7156169-be29-4e7b-b323-6f18e4e09ba9" containerName="payload-pre-processing"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781679    5282 cpu_manager.go:414] "containerMap: removing container" podUID="321af73b-d2a5-4e37-9ee6-a6f75d0b263c" containerName="payload-processing"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781688    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321af73b-d2a5-4e37-9ee6-a6f75d0b263c" containerName="payload-processing"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781702    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy"
Nov 03 20:26:04.782223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.781710    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy"
Nov 03 20:26:04.793409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.793370    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0972f4db-9691-4eb9-94a2-5dd461e656dd" containerName="maas-api"
Nov 03 20:26:04.793625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.793608    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe48dd6d-6155-474b-9cbd-01bc9898b66f" containerName="maas-api"
Nov 03 20:26:04.793737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.793725    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7156169-be29-4e7b-b323-6f18e4e09ba9" containerName="payload-pre-processing"
Nov 03 20:26:04.793845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.793835    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="321af73b-d2a5-4e37-9ee6-a6f75d0b263c" containerName="payload-processing"
Nov 03 20:26:04.793938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.793929    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f008b1c7-64d5-4ed3-8954-b789905c0f0e" containerName="istio-proxy"
Nov 03 20:26:04.794024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794015    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d8c32f-89d6-403e-ac8f-47fcf85f2632" containerName="main"
Nov 03 20:26:04.794102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794093    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="690ea14a-8ce1-4776-a478-fd9a2b66b4c9" containerName="istio-proxy"
Nov 03 20:26:04.794171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794163    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a4a27d-6002-43e8-bfa5-af8a681cbf5d" containerName="maas-api"
Nov 03 20:26:04.794279 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794267    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy"
Nov 03 20:26:04.794384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794371    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="17af1c04-f9ef-4e5b-808b-b1c15e9eb5f2" containerName="istio-proxy"
Nov 03 20:26:04.794495 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794484    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f42331-cba4-451d-b86c-89dc9e983332" containerName="istio-proxy"
Nov 03 20:26:04.794642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794630    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea21d06-e50b-47f7-9921-fe38f1c010d9" containerName="main"
Nov 03 20:26:04.794753 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794742    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy"
Nov 03 20:26:04.794829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794819    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy"
Nov 03 20:26:04.794916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794907    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ca8d9b-bda8-46c9-bb3e-69aa77e06445" containerName="payload-processing"
Nov 03 20:26:04.794997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.794986    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="de77257b-93d9-472b-a007-a6d199acf1e2" containerName="istio-proxy"
Nov 03 20:26:04.795076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.795067    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b272424b-a840-4c8f-9622-09136854de34" containerName="main"
Nov 03 20:26:04.835247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.818592    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.863097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.863028    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"]
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908225    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908303    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908349    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908390    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908463    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908628    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908662    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t444w\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-kube-api-access-t444w\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908706    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce92ce4-1f60-438c-8741-0c2238548df4-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:04.910039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:04.908848    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.055460    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.055563    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.055704    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.055734    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t444w\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-kube-api-access-t444w\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.055774    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce92ce4-1f60-438c-8741-0c2238548df4-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.055904    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.055985    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.056022    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.071290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.056058    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.073669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.073639    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce92ce4-1f60-438c-8741-0c2238548df4-istiod-ca-cert\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.074055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.074037    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-certs\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.074399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.074380    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.075027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.075006    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-credential-socket\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.075379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.075359    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-data\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.105038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.104941    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-envoy\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.107488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.107423    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-podinfo\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.108482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.108452    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-token\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.128274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.124226    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t444w\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-kube-api-access-t444w\") pod \"e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") " pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.184238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.183462    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:05.499573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.499526    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0972f4db-9691-4eb9-94a2-5dd461e656dd" path="/var/lib/kubelet/pods/0972f4db-9691-4eb9-94a2-5dd461e656dd/volumes"
Nov 03 20:26:05.706472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.706437    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:05.804015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.803845    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"]
Nov 03 20:26:05.813024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.812988    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:05.813260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.813234    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:05.813364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:05.813288    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:26:06.189396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.189255    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" event={"ID":"5ce92ce4-1f60-438c-8741-0c2238548df4","Type":"ContainerStarted","Data":"30c7116635c72ccf4537ef5734c250481ef9a684c70c8b11e48a85b302586a6c"}
Nov 03 20:26:06.189396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.189322    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" event={"ID":"5ce92ce4-1f60-438c-8741-0c2238548df4","Type":"ContainerStarted","Data":"6008f1778ddc46e9cf36794b34e5cba72c071cd237a9809cb1e1699000c4997c"}
Nov 03 20:26:06.216091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.216038    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" podStartSLOduration=2.216022818 podStartE2EDuration="2.216022818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:06.212144993 +0000 UTC m=+2758.881346462" watchObservedRunningTime="2026-11-03 20:26:06.216022818 +0000 UTC m=+2758.885224288"
Nov 03 20:26:06.296055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.295989    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:06.411586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.411534    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:06.533398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.533357    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"]
Nov 03 20:26:06.533636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.533611    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy" containerID="cri-o://a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa" gracePeriod=30
Nov 03 20:26:06.544564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.544536    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"]
Nov 03 20:26:06.549050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.549021    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"]
Nov 03 20:26:06.549421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.549391    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" containerID="cri-o://f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e" gracePeriod=30
Nov 03 20:26:06.585221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.582611    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:06.747952 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.747842    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:06.971708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:06.971624    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"]
Nov 03 20:26:07.007259 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.006603    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"]
Nov 03 20:26:07.096228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.090792    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.199553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.198431    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:07.199993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.199965    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:07.200068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.200028    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" podUID="5ce92ce4-1f60-438c-8741-0c2238548df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:07.206593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.206561    5282 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" secret="" err="secret \"e2e-shared-a-w0-cad531-openshift-default-dockercfg-gj4fh\" not found"
Nov 03 20:26:07.208787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.208762    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.231577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.231552    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"]
Nov 03 20:26:07.338851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.338821    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.344642 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.344534    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" podUID="ca057d53-559b-4e56-9635-24ef10b0b325" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:26:07.456525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.456493    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.607275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.594054    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:07.860476 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:07.860391    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt" podUID="bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" containerName="payload-pre-processing" containerID="cri-o://36aad6d9f4b1c19df0832a6cd7bc8539037852e7b511fc88de0052932ffa01e9" gracePeriod=25
Nov 03 20:26:08.140316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.136675    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 20:26:08.209227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.208694    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:26:08.209227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.208770    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" podUID="5ce92ce4-1f60-438c-8741-0c2238548df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Nov 03 20:26:08.223013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.221723    5282 generic.go:356] "Generic (PLEG): container finished" podID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerID="796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1" exitCode=0
Nov 03 20:26:08.223013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.221789    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"
Nov 03 20:26:08.223013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.221897    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" event={"ID":"ab4e8061-b105-45fc-8a9e-a35605b4bcb7","Type":"ContainerDied","Data":"796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1"}
Nov 03 20:26:08.223013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.221965    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt" event={"ID":"ab4e8061-b105-45fc-8a9e-a35605b4bcb7","Type":"ContainerDied","Data":"29904f8bcccf3e0d37ff906c45c7da804929af7e971a17a84d626e08786550e6"}
Nov 03 20:26:08.223013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.221990    5282 scope.go:122] "RemoveContainer" containerID="796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1"
Nov 03 20:26:08.225156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.224444    5282 generic.go:356] "Generic (PLEG): container finished" podID="bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" containerID="36aad6d9f4b1c19df0832a6cd7bc8539037852e7b511fc88de0052932ffa01e9" exitCode=0
Nov 03 20:26:08.225156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.224571    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" podUID="5ce92ce4-1f60-438c-8741-0c2238548df4" containerName="istio-proxy" containerID="cri-o://30c7116635c72ccf4537ef5734c250481ef9a684c70c8b11e48a85b302586a6c" gracePeriod=30
Nov 03 20:26:08.225156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.224782    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt" event={"ID":"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac","Type":"ContainerDied","Data":"36aad6d9f4b1c19df0832a6cd7bc8539037852e7b511fc88de0052932ffa01e9"}
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.227771    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-envoy\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.227828    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-podinfo\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.227878    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-data\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.227974    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-token\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-token\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.228010    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-kube-api-access-swwfd\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-kube-api-access-swwfd\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.228085    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-certs\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.228376    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-socket\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.228439    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-credential-socket\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.229329    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-certs" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.229678    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-socket" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.229995    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istiod-ca-cert\") pod \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\" (UID: \"ab4e8061-b105-45fc-8a9e-a35605b4bcb7\") "
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.230721    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.230744    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.230912    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-credential-socket" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.231086    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istiod-ca-cert" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:08.232177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.231266    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-data" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.234347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.234307    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-podinfo" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:08.236687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.236642    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-envoy" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:08.236798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.236761    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-kube-api-access-swwfd" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "kube-api-access-swwfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:08.237584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.237558    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-token" pod "ab4e8061-b105-45fc-8a9e-a35605b4bcb7" (UID: "ab4e8061-b105-45fc-8a9e-a35605b4bcb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:08.244393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.244373    5282 scope.go:122] "RemoveContainer" containerID="796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1"
Nov 03 20:26:08.244740 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:08.244715    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1\": container with ID starting with 796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1 not found: ID does not exist" containerID="796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1"
Nov 03 20:26:08.244812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.244756    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1"} err="failed to get container status \"796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1\": rpc error: code = NotFound desc = could not find container \"796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1\": container with ID starting with 796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1 not found: ID does not exist"
Nov 03 20:26:08.313946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.313920    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:26:08.336684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.332638    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.336684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.332682    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.336684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.332698    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.336684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.332712    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.336684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.332740    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.336684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.332755    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.336684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.332771    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwfd\" (UniqueName: \"kubernetes.io/projected/ab4e8061-b105-45fc-8a9e-a35605b4bcb7-kube-api-access-swwfd\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.434327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.434246    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-kube-api-access-s9g5j\" (UniqueName: \"kubernetes.io/projected/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-kube-api-access-s9g5j\") pod \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") "
Nov 03 20:26:08.434327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.434298    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-plugins-config-volume\") pod \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\" (UID: \"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac\") "
Nov 03 20:26:08.434682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.434654    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-plugins-config-volume" pod "bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" (UID: "bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:08.436439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.436419    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-kube-api-access-s9g5j" pod "bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" (UID: "bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac"). InnerVolumeSpecName "kube-api-access-s9g5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:08.536440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.536411    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9g5j\" (UniqueName: \"kubernetes.io/projected/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-kube-api-access-s9g5j\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.536573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.536443    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:08.549557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.549526    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"]
Nov 03 20:26:08.552579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.552545    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kpkjt"]
Nov 03 20:26:08.799391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.799357    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:08.945419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.945385    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:08.945568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:08.945447    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:09.234885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.234853    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt" event={"ID":"bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac","Type":"ContainerDied","Data":"d51c833745269aa1686221eb672db5c5cf7f48e79d1147694834f8f8e679b86b"}
Nov 03 20:26:09.234885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.234882    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"
Nov 03 20:26:09.235453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.234906    5282 scope.go:122] "RemoveContainer" containerID="36aad6d9f4b1c19df0832a6cd7bc8539037852e7b511fc88de0052932ffa01e9"
Nov 03 20:26:09.263746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.263717    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"]
Nov 03 20:26:09.268898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.268869    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-0ffd89-6cb465c8c9-m7rlt"]
Nov 03 20:26:09.456594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.456550    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" path="/var/lib/kubelet/pods/ab4e8061-b105-45fc-8a9e-a35605b4bcb7/volumes"
Nov 03 20:26:09.457014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.456997    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" path="/var/lib/kubelet/pods/bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac/volumes"
Nov 03 20:26:09.503018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.502932    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:09.699222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.698675    5282 scope.go:122] "RemoveContainer" containerID="45232dcff46a1f8a028feb3e5aa4a00283603068d93c54bf4d80280c73569ff3"
Nov 03 20:26:09.715036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.715010    5282 scope.go:122] "RemoveContainer" containerID="7cd3d9a082741581f3f6645eddf50c2fe6f494a0f767f35166bb2a22f7f24d68"
Nov 03 20:26:09.726973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.725341    5282 scope.go:122] "RemoveContainer" containerID="8b941d55abe35057078c71f99a74fbc915cdc240d61079ade03fc40f2c4fad61"
Nov 03 20:26:09.736605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:09.736577    5282 scope.go:122] "RemoveContainer" containerID="31df442ff7e6f3b98ff956bee23e7596443ca2ffba45e157784bfd876e8290ad"
Nov 03 20:26:10.211119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:10.211019    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:10.563576 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:10.563539    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:10.693690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:10.693618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:10.818438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:10.818328    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"]
Nov 03 20:26:10.819520 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:10.819290    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy" containerID="cri-o://7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c" gracePeriod=30
Nov 03 20:26:10.823528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:10.823502    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:10.974221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:10.971857    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:11.352973 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:11.352939    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:11.932227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:11.930341    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:11.990472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:11.974780    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq" podUID="abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" containerName="payload-processing" containerID="cri-o://afb46bcc5ffda08cd1ac2fca2c0e89278a02009c07fc2956d059cc312ceee774" gracePeriod=25
Nov 03 20:26:12.005736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.005384    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:26:12.007293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.007134    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh" podUID="03dece0c-cf3c-4d61-833d-a7a8a3531666" containerName="payload-pre-processing" containerID="cri-o://b29d786134db65e08463275384b6591b4eabc12250eb6310c5d4a5e01b0c4aa9" gracePeriod=25
Nov 03 20:26:12.125271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125236    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-certs\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125319    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-kube-api-access-xbw64\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-kube-api-access-xbw64\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125418    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-data\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125464    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-socket\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125521    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-envoy\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125533    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-certs" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.125611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125553    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-credential-socket\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125629    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-token\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-token\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125702    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-data" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.125769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125750    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-podinfo\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125802    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istiod-ca-cert\") pod \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\" (UID: \"f777c24d-c5bf-4e9e-a65f-eef2aa170470\") "
Nov 03 20:26:12.125929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.125877    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-socket" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.126893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.126769    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.126893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.126798    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.126893 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.126815    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.127331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.127191    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istiod-ca-cert" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.127842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.127817    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-credential-socket" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.128552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.128221    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-token" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.128552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.128358    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-podinfo" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:12.128552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.128377    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-kube-api-access-xbw64" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "kube-api-access-xbw64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.128811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.128794    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-envoy" pod "f777c24d-c5bf-4e9e-a65f-eef2aa170470" (UID: "f777c24d-c5bf-4e9e-a65f-eef2aa170470"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.178280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.173823    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:26:12.228770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.228723    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e6368c-b747-4898-bed9-6d4e60dda094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e6368c-b747-4898-bed9-6d4e60dda094-istiod-ca-cert\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.228955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.228802    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-podinfo\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.228955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.228905    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-certs\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.228955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.228935    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-envoy\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.229246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.228980    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-credential-socket\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.229246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.229092    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-token\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-token\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.229350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.229278    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-socket\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.229350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.229326    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-kube-api-access-vvnzg\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-kube-api-access-vvnzg\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.229473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.229378    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-data\") pod \"c4e6368c-b747-4898-bed9-6d4e60dda094\" (UID: \"c4e6368c-b747-4898-bed9-6d4e60dda094\") "
Nov 03 20:26:12.230757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.230731    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e6368c-b747-4898-bed9-6d4e60dda094-istiod-ca-cert" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.230929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.230913    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-certs" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.231646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.231618    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-socket" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.231909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.231875    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-credential-socket" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.232028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.231958    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerName="payload-processing" containerID="cri-o://72a6485d80472dea967a4cc228bffdd476cf5f446a5f34841be9fe5049bf8f60" gracePeriod=25
Nov 03 20:26:12.233131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.233101    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-data" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.234047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.234019    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-kube-api-access-vvnzg" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "kube-api-access-vvnzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.234176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.234137    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-envoy" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:12.235221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235171    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235303    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235328    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235344    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235375    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235387    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235401    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbw64\" (UniqueName: \"kubernetes.io/projected/f777c24d-c5bf-4e9e-a65f-eef2aa170470-kube-api-access-xbw64\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235416    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235449    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvnzg\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-kube-api-access-vvnzg\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235488    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235518    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4e6368c-b747-4898-bed9-6d4e60dda094-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235533    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.235603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.235548    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f777c24d-c5bf-4e9e-a65f-eef2aa170470-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.237358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.237319    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-podinfo" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:12.237358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.237328    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-token" pod "c4e6368c-b747-4898-bed9-6d4e60dda094" (UID: "c4e6368c-b747-4898-bed9-6d4e60dda094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.285134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.285093    5282 generic.go:356] "Generic (PLEG): container finished" podID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerID="a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa" exitCode=0
Nov 03 20:26:12.285403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.285265    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" event={"ID":"c4e6368c-b747-4898-bed9-6d4e60dda094","Type":"ContainerDied","Data":"a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa"}
Nov 03 20:26:12.285403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.285358    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj" event={"ID":"c4e6368c-b747-4898-bed9-6d4e60dda094","Type":"ContainerDied","Data":"9eb2783401b17153f0038dfabda5c1f8d97af9bcec76fd9c77bee6a9f37ef933"}
Nov 03 20:26:12.285403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.285392    5282 scope.go:122] "RemoveContainer" containerID="a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa"
Nov 03 20:26:12.285623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.285537    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"
Nov 03 20:26:12.289411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.289382    5282 generic.go:356] "Generic (PLEG): container finished" podID="03dece0c-cf3c-4d61-833d-a7a8a3531666" containerID="b29d786134db65e08463275384b6591b4eabc12250eb6310c5d4a5e01b0c4aa9" exitCode=0
Nov 03 20:26:12.289582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.289558    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh" event={"ID":"03dece0c-cf3c-4d61-833d-a7a8a3531666","Type":"ContainerDied","Data":"b29d786134db65e08463275384b6591b4eabc12250eb6310c5d4a5e01b0c4aa9"}
Nov 03 20:26:12.291464 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.291430    5282 generic.go:356] "Generic (PLEG): container finished" podID="ca057d53-559b-4e56-9635-24ef10b0b325" containerID="f80cdbc0aadd09a85a75dba987b8865421f4f6dc5e36f28fdab7bc8d4c3b8133" exitCode=0
Nov 03 20:26:12.291559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.291505    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" event={"ID":"ca057d53-559b-4e56-9635-24ef10b0b325","Type":"ContainerDied","Data":"f80cdbc0aadd09a85a75dba987b8865421f4f6dc5e36f28fdab7bc8d4c3b8133"}
Nov 03 20:26:12.293190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.293135    5282 generic.go:356] "Generic (PLEG): container finished" podID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerID="f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e" exitCode=0
Nov 03 20:26:12.293278 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.293228    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"
Nov 03 20:26:12.293394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.293371    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" event={"ID":"f777c24d-c5bf-4e9e-a65f-eef2aa170470","Type":"ContainerDied","Data":"f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e"}
Nov 03 20:26:12.293533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.293510    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w" event={"ID":"f777c24d-c5bf-4e9e-a65f-eef2aa170470","Type":"ContainerDied","Data":"0474886e5cfcefe45029b7207ab28b578cd289cd7c3f21857554ad8c4c9c4857"}
Nov 03 20:26:12.296573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.296552    5282 generic.go:356] "Generic (PLEG): container finished" podID="abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" containerID="afb46bcc5ffda08cd1ac2fca2c0e89278a02009c07fc2956d059cc312ceee774" exitCode=0
Nov 03 20:26:12.296730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.296618    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq" event={"ID":"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3","Type":"ContainerDied","Data":"afb46bcc5ffda08cd1ac2fca2c0e89278a02009c07fc2956d059cc312ceee774"}
Nov 03 20:26:12.303221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.302986    5282 scope.go:122] "RemoveContainer" containerID="a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa"
Nov 03 20:26:12.304065 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:12.303412    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa\": container with ID starting with a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa not found: ID does not exist" containerID="a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa"
Nov 03 20:26:12.304065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.303440    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa"} err="failed to get container status \"a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa\": rpc error: code = NotFound desc = could not find container \"a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa\": container with ID starting with a9ffef2445db6ccae6c3cbaee7e1d3857abbd9762159d885f0254f1ff838ebaa not found: ID does not exist"
Nov 03 20:26:12.304065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.303456    5282 scope.go:122] "RemoveContainer" containerID="f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e"
Nov 03 20:26:12.323421 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.323382    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"]
Nov 03 20:26:12.325254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.325218    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-mtzrj"]
Nov 03 20:26:12.327987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.327966    5282 scope.go:122] "RemoveContainer" containerID="f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e"
Nov 03 20:26:12.329180 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:12.329157    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e\": container with ID starting with f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e not found: ID does not exist" containerID="f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e"
Nov 03 20:26:12.329307 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.329186    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e"} err="failed to get container status \"f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e\": rpc error: code = NotFound desc = could not find container \"f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e\": container with ID starting with f2dca74f642f3862fa3ec1e565d2309b2073be220d0c06449ad12f818e38cf3e not found: ID does not exist"
Nov 03 20:26:12.336593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.336568    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"]
Nov 03 20:26:12.337416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.337395    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.337416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.337416    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4e6368c-b747-4898-bed9-6d4e60dda094-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.342852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.342824    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-n8t7w"]
Nov 03 20:26:12.345223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.345187    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:26:12.412019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.411991    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:26:12.416657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.416635    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:26:12.438511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.438477    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-kube-api-access-sc5w9\" (UniqueName: \"kubernetes.io/projected/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-kube-api-access-sc5w9\") pod \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") "
Nov 03 20:26:12.438715 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.438693    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-plugins-config-volume\") pod \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\" (UID: \"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3\") "
Nov 03 20:26:12.438791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.438734    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:26:12.439459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.439431    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-plugins-config-volume" pod "abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" (UID: "abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.441381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.441354    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-kube-api-access-sc5w9" pod "abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" (UID: "abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3"). InnerVolumeSpecName "kube-api-access-sc5w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.540314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.540281    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03dece0c-cf3c-4d61-833d-a7a8a3531666-kube-api-access-7blkn\" (UniqueName: \"kubernetes.io/projected/03dece0c-cf3c-4d61-833d-a7a8a3531666-kube-api-access-7blkn\") pod \"03dece0c-cf3c-4d61-833d-a7a8a3531666\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") "
Nov 03 20:26:12.540467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.540394    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03dece0c-cf3c-4d61-833d-a7a8a3531666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03dece0c-cf3c-4d61-833d-a7a8a3531666-plugins-config-volume\") pod \"03dece0c-cf3c-4d61-833d-a7a8a3531666\" (UID: \"03dece0c-cf3c-4d61-833d-a7a8a3531666\") "
Nov 03 20:26:12.540531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.540520    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca057d53-559b-4e56-9635-24ef10b0b325-kube-api-access-fjzgx\" (UniqueName: \"kubernetes.io/projected/ca057d53-559b-4e56-9635-24ef10b0b325-kube-api-access-fjzgx\") pod \"ca057d53-559b-4e56-9635-24ef10b0b325\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") "
Nov 03 20:26:12.540610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.540595    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca057d53-559b-4e56-9635-24ef10b0b325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca057d53-559b-4e56-9635-24ef10b0b325-plugins-config-volume\") pod \"ca057d53-559b-4e56-9635-24ef10b0b325\" (UID: \"ca057d53-559b-4e56-9635-24ef10b0b325\") "
Nov 03 20:26:12.540910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.540884    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03dece0c-cf3c-4d61-833d-a7a8a3531666-plugins-config-volume" pod "03dece0c-cf3c-4d61-833d-a7a8a3531666" (UID: "03dece0c-cf3c-4d61-833d-a7a8a3531666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.541006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.540986    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca057d53-559b-4e56-9635-24ef10b0b325-plugins-config-volume" pod "ca057d53-559b-4e56-9635-24ef10b0b325" (UID: "ca057d53-559b-4e56-9635-24ef10b0b325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:12.541139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.541123    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca057d53-559b-4e56-9635-24ef10b0b325-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.541224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.541140    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.541224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.541150    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03dece0c-cf3c-4d61-833d-a7a8a3531666-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.541224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.541159    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5w9\" (UniqueName: \"kubernetes.io/projected/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3-kube-api-access-sc5w9\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.542386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.542362    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dece0c-cf3c-4d61-833d-a7a8a3531666-kube-api-access-7blkn" pod "03dece0c-cf3c-4d61-833d-a7a8a3531666" (UID: "03dece0c-cf3c-4d61-833d-a7a8a3531666"). InnerVolumeSpecName "kube-api-access-7blkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.542630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.542612    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca057d53-559b-4e56-9635-24ef10b0b325-kube-api-access-fjzgx" pod "ca057d53-559b-4e56-9635-24ef10b0b325" (UID: "ca057d53-559b-4e56-9635-24ef10b0b325"). InnerVolumeSpecName "kube-api-access-fjzgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:12.642845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.642805    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjzgx\" (UniqueName: \"kubernetes.io/projected/ca057d53-559b-4e56-9635-24ef10b0b325-kube-api-access-fjzgx\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.642845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.642840    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7blkn\" (UniqueName: \"kubernetes.io/projected/03dece0c-cf3c-4d61-833d-a7a8a3531666-kube-api-access-7blkn\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:12.662635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.662552    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:12.811551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.811513    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:12.962274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:12.961784    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:13.191034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.191005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:13.305612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.305584    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"
Nov 03 20:26:13.305612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.305607    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq" event={"ID":"abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3","Type":"ContainerDied","Data":"3c265724397d866ed10ce13f7fba972b09ac0a167b4d965b5e2835747276d202"}
Nov 03 20:26:13.305854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.305670    5282 scope.go:122] "RemoveContainer" containerID="afb46bcc5ffda08cd1ac2fca2c0e89278a02009c07fc2956d059cc312ceee774"
Nov 03 20:26:13.308393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.308361    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh" event={"ID":"03dece0c-cf3c-4d61-833d-a7a8a3531666","Type":"ContainerDied","Data":"346d031549f8748a7c7d9aeb72f65ecd42a740aa3c021d00c89cce91477aeea0"}
Nov 03 20:26:13.308393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.308386    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"
Nov 03 20:26:13.310353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.310258    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm" event={"ID":"ca057d53-559b-4e56-9635-24ef10b0b325","Type":"ContainerDied","Data":"5b0072ee42eb8cec0a8d695b9c8eb29e054fd89b7c41315698bb1f6cffc9eb7f"}
Nov 03 20:26:13.310353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.310275    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"
Nov 03 20:26:13.318855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.318837    5282 scope.go:122] "RemoveContainer" containerID="b29d786134db65e08463275384b6591b4eabc12250eb6310c5d4a5e01b0c4aa9"
Nov 03 20:26:13.328173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.328152    5282 scope.go:122] "RemoveContainer" containerID="f80cdbc0aadd09a85a75dba987b8865421f4f6dc5e36f28fdab7bc8d4c3b8133"
Nov 03 20:26:13.398217 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.395000    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"]
Nov 03 20:26:13.400825 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.400739    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d81d18-5f89dd9858-hb8mq"]
Nov 03 20:26:13.409302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.409255    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"]
Nov 03 20:26:13.421258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.416963    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d81d18-6448f949ff-pksjh"]
Nov 03 20:26:13.428496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.428443    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"]
Nov 03 20:26:13.436765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.434294    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0a7343-cf5db545f-tp5vm"]
Nov 03 20:26:13.441436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.441400    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:13.455867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.455838    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03dece0c-cf3c-4d61-833d-a7a8a3531666" path="/var/lib/kubelet/pods/03dece0c-cf3c-4d61-833d-a7a8a3531666/volumes"
Nov 03 20:26:13.456348 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.456326    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" path="/var/lib/kubelet/pods/abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3/volumes"
Nov 03 20:26:13.456790 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.456775    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" path="/var/lib/kubelet/pods/c4e6368c-b747-4898-bed9-6d4e60dda094/volumes"
Nov 03 20:26:13.457482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.457467    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca057d53-559b-4e56-9635-24ef10b0b325" path="/var/lib/kubelet/pods/ca057d53-559b-4e56-9635-24ef10b0b325/volumes"
Nov 03 20:26:13.457913 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.457899    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" path="/var/lib/kubelet/pods/f777c24d-c5bf-4e9e-a65f-eef2aa170470/volumes"
Nov 03 20:26:13.561030 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:13.560953    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:13.731446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:13.731414    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:14.093572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:14.093540    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:14.749130 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:14.749091    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:14.776293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:14.776256    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:15.404658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:15.404614    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:16.045556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.045524    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:16.163446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.163417    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:26:16.297378 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297347    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-token\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-token\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.297556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297396    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-data\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.297556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297521    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istiod-ca-cert\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.297659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297640    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-credential-socket\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.297702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297690    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-envoy\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.297760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297713    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-data" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:16.297820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297765    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-podinfo\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.297884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297826    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-certs\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.297947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297878    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-credential-socket" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:16.297947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297887    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istiod-ca-cert" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:16.297947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297917    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-socket\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.298108 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.297966    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-kube-api-access-swvr5\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-kube-api-access-swvr5\") pod \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\" (UID: \"f48c0353-24d5-43a4-ba99-ee8d666ad6a2\") "
Nov 03 20:26:16.298166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.298153    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-socket" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:16.298253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.298171    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-certs" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:16.298512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.298494    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.298605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.298514    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.298605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.298523    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.298605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.298531    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.298605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.298540    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.299917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.299893    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-podinfo" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:16.300132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.300105    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-kube-api-access-swvr5" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "kube-api-access-swvr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:16.300132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.300118    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-envoy" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:16.300132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.300118    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-token" pod "f48c0353-24d5-43a4-ba99-ee8d666ad6a2" (UID: "f48c0353-24d5-43a4-ba99-ee8d666ad6a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:16.337222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.337162    5282 generic.go:356] "Generic (PLEG): container finished" podID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerID="7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c" exitCode=0
Nov 03 20:26:16.337357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.337229    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" event={"ID":"f48c0353-24d5-43a4-ba99-ee8d666ad6a2","Type":"ContainerDied","Data":"7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c"}
Nov 03 20:26:16.337357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.337292    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc" event={"ID":"f48c0353-24d5-43a4-ba99-ee8d666ad6a2","Type":"ContainerDied","Data":"f9186da31ae65b5f35bc09d0931ee35d199264820389c6a4a2182104174d0107"}
Nov 03 20:26:16.337357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.337315    5282 scope.go:122] "RemoveContainer" containerID="7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c"
Nov 03 20:26:16.337357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.337243    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"
Nov 03 20:26:16.348306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.348289    5282 scope.go:122] "RemoveContainer" containerID="7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c"
Nov 03 20:26:16.348604 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:16.348582    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c\": container with ID starting with 7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c not found: ID does not exist" containerID="7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c"
Nov 03 20:26:16.348658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.348612    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c"} err="failed to get container status \"7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c\": rpc error: code = NotFound desc = could not find container \"7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c\": container with ID starting with 7b0404b80a1e2f4a352bc9876a71aadee4285a3a26cafb855707adfc363f799c not found: ID does not exist"
Nov 03 20:26:16.365528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.365504    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"]
Nov 03 20:26:16.370933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.370912    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-0ffd89-openshift-default-6d49fd96c5-6fnhc"]
Nov 03 20:26:16.399509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.399459    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvr5\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-kube-api-access-swvr5\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.399509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.399481    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.399509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.399491    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.399509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.399502    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48c0353-24d5-43a4-ba99-ee8d666ad6a2-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:16.415072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.415040    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:16.904901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:16.903963    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:17.120557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.120523    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"]
Nov 03 20:26:17.214431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.214391    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:17.341754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.341703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:17.349793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.349763    5282 generic.go:356] "Generic (PLEG): container finished" podID="5ce92ce4-1f60-438c-8741-0c2238548df4" containerID="30c7116635c72ccf4537ef5734c250481ef9a684c70c8b11e48a85b302586a6c" exitCode=0
Nov 03 20:26:17.349932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.349842    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" event={"ID":"5ce92ce4-1f60-438c-8741-0c2238548df4","Type":"ContainerDied","Data":"30c7116635c72ccf4537ef5734c250481ef9a684c70c8b11e48a85b302586a6c"}
Nov 03 20:26:17.545218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.531386    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" path="/var/lib/kubelet/pods/f48c0353-24d5-43a4-ba99-ee8d666ad6a2/volumes"
Nov 03 20:26:17.570430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.570404    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:17.725105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725065    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-socket\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725168    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-data\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725213    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-envoy\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725287    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-credential-socket\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725369    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-socket" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:17.725528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725397    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce92ce4-1f60-438c-8741-0c2238548df4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce92ce4-1f60-438c-8741-0c2238548df4-istiod-ca-cert\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725528 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725470    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-data" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:17.725692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725525    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-kube-api-access-t444w\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-kube-api-access-t444w\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725561    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-podinfo\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725595    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-token\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-token\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725637    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-certs\") pod \"5ce92ce4-1f60-438c-8741-0c2238548df4\" (UID: \"5ce92ce4-1f60-438c-8741-0c2238548df4\") "
Nov 03 20:26:17.725888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725593    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-credential-socket" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:17.725888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.725715    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce92ce4-1f60-438c-8741-0c2238548df4-istiod-ca-cert" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:17.726043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.726021    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-certs" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:17.726478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.726454    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.726545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.726482    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.726545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.726498    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ce92ce4-1f60-438c-8741-0c2238548df4-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.726545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.726512    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.726545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.726529    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.727801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.727774    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-podinfo" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:17.727996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.727967    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-kube-api-access-t444w" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "kube-api-access-t444w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:17.728057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.728003    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-token" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:17.728057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.728031    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-envoy" pod "5ce92ce4-1f60-438c-8741-0c2238548df4" (UID: "5ce92ce4-1f60-438c-8741-0c2238548df4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:17.806691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.806619    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:17.839589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.836234    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t444w\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-kube-api-access-t444w\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.839589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.836277    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.839589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.836293    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:17.839589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:17.836306    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ce92ce4-1f60-438c-8741-0c2238548df4-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:18.357033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:18.357001    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"
Nov 03 20:26:18.357242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:18.357045    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc" event={"ID":"5ce92ce4-1f60-438c-8741-0c2238548df4","Type":"ContainerDied","Data":"6008f1778ddc46e9cf36794b34e5cba72c071cd237a9809cb1e1699000c4997c"}
Nov 03 20:26:18.357242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:18.357111    5282 scope.go:122] "RemoveContainer" containerID="30c7116635c72ccf4537ef5734c250481ef9a684c70c8b11e48a85b302586a6c"
Nov 03 20:26:18.385304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:18.385272    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"]
Nov 03 20:26:18.390048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:18.390024    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cad531-openshift-default-6bbdbd8f65-97xwc"]
Nov 03 20:26:18.482172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:18.482137    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:19.279453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:19.279409    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:19.455228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:19.455176    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce92ce4-1f60-438c-8741-0c2238548df4" path="/var/lib/kubelet/pods/5ce92ce4-1f60-438c-8741-0c2238548df4/volumes"
Nov 03 20:26:19.864911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:19.864880    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:19.984462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:19.984427    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:20.130026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.129944    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"]
Nov 03 20:26:20.130499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.130464    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy" containerID="cri-o://5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26" gracePeriod=30
Nov 03 20:26:20.204069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.204036    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:20.330665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.330632    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:20.676355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.676324    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:20.797270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.797231    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:20.919004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.918972    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"]
Nov 03 20:26:20.919273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.919206    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" containerID="cri-o://e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d" gracePeriod=30
Nov 03 20:26:20.973998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:20.973912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:21.105941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:21.105877    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:21.734011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:21.733969    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"]
Nov 03 20:26:21.838773 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:21.838710    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:21.965154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:21.965124    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:22.092087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.091996    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:22.123363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.123322    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd" podUID="68cfe518-7312-4bbb-8700-f8a165a77251" containerName="payload-processing" containerID="cri-o://c1944a589cede9a8524a14036b574e5ee655400e60cb4454341c78e27e9523ba" gracePeriod=25
Nov 03 20:26:22.178677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.178640    5282 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:22.178824 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.178703    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:22.400577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.400542    5282 generic.go:356] "Generic (PLEG): container finished" podID="68cfe518-7312-4bbb-8700-f8a165a77251" containerID="c1944a589cede9a8524a14036b574e5ee655400e60cb4454341c78e27e9523ba" exitCode=0
Nov 03 20:26:22.400694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.400615    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd" event={"ID":"68cfe518-7312-4bbb-8700-f8a165a77251","Type":"ContainerDied","Data":"c1944a589cede9a8524a14036b574e5ee655400e60cb4454341c78e27e9523ba"}
Nov 03 20:26:22.400734 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.400715    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd" event={"ID":"68cfe518-7312-4bbb-8700-f8a165a77251","Type":"ContainerDied","Data":"9e063c7538673dd2f0d8940aab01b3e3ab28f1fb12e923d8bcd6d07709f8c56a"}
Nov 03 20:26:22.400774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.400739    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e063c7538673dd2f0d8940aab01b3e3ab28f1fb12e923d8bcd6d07709f8c56a"
Nov 03 20:26:22.417703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.417679    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:26:22.435363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.435325    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:26:22.580815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.580779    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cfe518-7312-4bbb-8700-f8a165a77251-kube-api-access-nrn8w\" (UniqueName: \"kubernetes.io/projected/68cfe518-7312-4bbb-8700-f8a165a77251-kube-api-access-nrn8w\") pod \"68cfe518-7312-4bbb-8700-f8a165a77251\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") "
Nov 03 20:26:22.580991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.580876    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68cfe518-7312-4bbb-8700-f8a165a77251-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cfe518-7312-4bbb-8700-f8a165a77251-plugins-config-volume\") pod \"68cfe518-7312-4bbb-8700-f8a165a77251\" (UID: \"68cfe518-7312-4bbb-8700-f8a165a77251\") "
Nov 03 20:26:22.581257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.581237    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cfe518-7312-4bbb-8700-f8a165a77251-plugins-config-volume" pod "68cfe518-7312-4bbb-8700-f8a165a77251" (UID: "68cfe518-7312-4bbb-8700-f8a165a77251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:22.581395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.581381    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68cfe518-7312-4bbb-8700-f8a165a77251-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:22.582949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.582929    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cfe518-7312-4bbb-8700-f8a165a77251-kube-api-access-nrn8w" pod "68cfe518-7312-4bbb-8700-f8a165a77251" (UID: "68cfe518-7312-4bbb-8700-f8a165a77251"). InnerVolumeSpecName "kube-api-access-nrn8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:22.640740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.640661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:22.694220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:22.688435    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrn8w\" (UniqueName: \"kubernetes.io/projected/68cfe518-7312-4bbb-8700-f8a165a77251-kube-api-access-nrn8w\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:23.168328 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:23.168300    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:23.407009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:23.406980    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"
Nov 03 20:26:23.408541 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:23.408510    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:23.432260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:23.432180    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"]
Nov 03 20:26:23.505605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:23.505561    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a476f8-77865dd486-k8fzd"]
Nov 03 20:26:23.565941 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:23.565896    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:23.579492 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:23.579394    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:23.821069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:23.821038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:24.686782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:24.686744    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:25.406964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.406900    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:26:25.421539 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.421511    5282 generic.go:356] "Generic (PLEG): container finished" podID="5a030b91-7598-43b4-8554-ba348887cf30" containerID="5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26" exitCode=0
Nov 03 20:26:25.421665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.421557    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"
Nov 03 20:26:25.421665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.421603    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" event={"ID":"5a030b91-7598-43b4-8554-ba348887cf30","Type":"ContainerDied","Data":"5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26"}
Nov 03 20:26:25.421665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.421653    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw" event={"ID":"5a030b91-7598-43b4-8554-ba348887cf30","Type":"ContainerDied","Data":"853f18ee3b52e6aae1a6ef5a407becaf87e9156478117aee241c425d32afd140"}
Nov 03 20:26:25.421779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.421681    5282 scope.go:122] "RemoveContainer" containerID="5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26"
Nov 03 20:26:25.432897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.432869    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-data\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433005    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-socket\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433072    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a030b91-7598-43b4-8554-ba348887cf30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a030b91-7598-43b4-8554-ba348887cf30-istio-podinfo\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433101    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-credential-socket\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433130    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-certs\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433178    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-istio-token\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-istio-token\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433270    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a030b91-7598-43b4-8554-ba348887cf30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a030b91-7598-43b4-8554-ba348887cf30-istiod-ca-cert\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433307    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-envoy\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433364    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-kube-api-access-xnbfm\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-kube-api-access-xnbfm\") pod \"5a030b91-7598-43b4-8554-ba348887cf30\" (UID: \"5a030b91-7598-43b4-8554-ba348887cf30\") "
Nov 03 20:26:25.433471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433371    5282 scope.go:122] "RemoveContainer" containerID="7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22"
Nov 03 20:26:25.433664 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433639    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-credential-socket" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.433736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.433667    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-certs" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.434037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.434018    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.434112 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.434044    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.434789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.434763    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a030b91-7598-43b4-8554-ba348887cf30-istiod-ca-cert" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:25.435162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.435137    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-socket" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.435381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.435361    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-data" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.438885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.438856    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-envoy" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.438990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.438945    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-kube-api-access-xnbfm" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "kube-api-access-xnbfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:25.438990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.438966    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-istio-token" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:25.439107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.439079    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a030b91-7598-43b4-8554-ba348887cf30-istio-podinfo" pod "5a030b91-7598-43b4-8554-ba348887cf30" (UID: "5a030b91-7598-43b4-8554-ba348887cf30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:25.491282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.450524    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:25.501113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.501075    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cfe518-7312-4bbb-8700-f8a165a77251" path="/var/lib/kubelet/pods/68cfe518-7312-4bbb-8700-f8a165a77251/volumes"
Nov 03 20:26:25.511608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.511573    5282 scope.go:122] "RemoveContainer" containerID="5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26"
Nov 03 20:26:25.511996 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:25.511975    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26\": container with ID starting with 5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26 not found: ID does not exist" containerID="5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26"
Nov 03 20:26:25.512076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.512007    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26"} err="failed to get container status \"5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26\": rpc error: code = NotFound desc = could not find container \"5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26\": container with ID starting with 5a8573f24c83a5bd2644c334baf2a9d7b68c4ab2a19db763f1aee93ef5a82f26 not found: ID does not exist"
Nov 03 20:26:25.512076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.512026    5282 scope.go:122] "RemoveContainer" containerID="7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22"
Nov 03 20:26:25.512373 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:25.512350    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22\": container with ID starting with 7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22 not found: ID does not exist" containerID="7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22"
Nov 03 20:26:25.512454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.512382    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22"} err="failed to get container status \"7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22\": rpc error: code = NotFound desc = could not find container \"7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22\": container with ID starting with 7efceb206f4488cf82a852f63706b6c03e080946db8f7310f63e85dae0e78b22 not found: ID does not exist"
Nov 03 20:26:25.535529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.535502    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.535529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.535527    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a030b91-7598-43b4-8554-ba348887cf30-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.535677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.535538    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.535677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.535546    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a030b91-7598-43b4-8554-ba348887cf30-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.535677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.535556    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.535677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.535566    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbfm\" (UniqueName: \"kubernetes.io/projected/5a030b91-7598-43b4-8554-ba348887cf30-kube-api-access-xnbfm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.535677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.535574    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a030b91-7598-43b4-8554-ba348887cf30-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.742774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.742740    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"]
Nov 03 20:26:25.746538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:25.746510    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-22a567-openshift-default-ff7c86f85-ml6fw"]
Nov 03 20:26:26.182271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.182250    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:26:26.230618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.230576    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.261695    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-data\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.261814    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-socket\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.261858    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-certs\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.261929    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-kube-api-access-zgthg\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-kube-api-access-zgthg\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.261960    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-envoy\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.262048    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfbfc761-e296-4114-b4e3-da2df1301934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbfc761-e296-4114-b4e3-da2df1301934-istio-podinfo\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.262120    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-credential-socket\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.262234    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-istio-token\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-istio-token\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.268148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.262269    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbfc761-e296-4114-b4e3-da2df1301934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbfc761-e296-4114-b4e3-da2df1301934-istiod-ca-cert\") pod \"dfbfc761-e296-4114-b4e3-da2df1301934\" (UID: \"dfbfc761-e296-4114-b4e3-da2df1301934\") "
Nov 03 20:26:26.270672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.270584    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbfc761-e296-4114-b4e3-da2df1301934-istiod-ca-cert" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.270989    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-data" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.271162    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-socket" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.271347    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-certs" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.272631    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.272664    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.272682    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.272695    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfbfc761-e296-4114-b4e3-da2df1301934-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.273019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.272984    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-credential-socket" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.296709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.296392    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfbfc761-e296-4114-b4e3-da2df1301934-istio-podinfo" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:26.304978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.304906    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-istio-token" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.305109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.305000    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-envoy" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:26.307299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.307268    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-kube-api-access-zgthg" pod "dfbfc761-e296-4114-b4e3-da2df1301934" (UID: "dfbfc761-e296-4114-b4e3-da2df1301934"). InnerVolumeSpecName "kube-api-access-zgthg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.374481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.374453    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfbfc761-e296-4114-b4e3-da2df1301934-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.374481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.374480    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.374669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.374490    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.374669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.374499    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgthg\" (UniqueName: \"kubernetes.io/projected/dfbfc761-e296-4114-b4e3-da2df1301934-kube-api-access-zgthg\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.374669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.374509    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfbfc761-e296-4114-b4e3-da2df1301934-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.430275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.430247    5282 generic.go:356] "Generic (PLEG): container finished" podID="dfbfc761-e296-4114-b4e3-da2df1301934" containerID="e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d" exitCode=0
Nov 03 20:26:26.430402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.430302    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"
Nov 03 20:26:26.430402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.430324    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" event={"ID":"dfbfc761-e296-4114-b4e3-da2df1301934","Type":"ContainerDied","Data":"e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d"}
Nov 03 20:26:26.430402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.430377    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq" event={"ID":"dfbfc761-e296-4114-b4e3-da2df1301934","Type":"ContainerDied","Data":"1f0d4133a98caf6d6b091998f7bf25b640d31fcfcf9e62749073e971f87dc41c"}
Nov 03 20:26:26.430402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.430401    5282 scope.go:122] "RemoveContainer" containerID="e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d"
Nov 03 20:26:26.441571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.441544    5282 scope.go:122] "RemoveContainer" containerID="1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5"
Nov 03 20:26:26.453375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.453353    5282 scope.go:122] "RemoveContainer" containerID="e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d"
Nov 03 20:26:26.453958 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:26.453938    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d\": container with ID starting with e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d not found: ID does not exist" containerID="e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d"
Nov 03 20:26:26.454025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.453966    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d"} err="failed to get container status \"e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d\": rpc error: code = NotFound desc = could not find container \"e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d\": container with ID starting with e2d4645653491c842add2224a8e815e6374549c6ee5797221a33548ef5ccbc6d not found: ID does not exist"
Nov 03 20:26:26.454025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.453983    5282 scope.go:122] "RemoveContainer" containerID="1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5"
Nov 03 20:26:26.454256 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:26.454236    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5\": container with ID starting with 1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5 not found: ID does not exist" containerID="1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5"
Nov 03 20:26:26.454303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.454263    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5"} err="failed to get container status \"1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5\": rpc error: code = NotFound desc = could not find container \"1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5\": container with ID starting with 1bf5bc9ebb282b55ef32c081a646494c1efeba349d50a26a3cec49799f1739d5 not found: ID does not exist"
Nov 03 20:26:26.458685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.458664    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"]
Nov 03 20:26:26.462968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.462948    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d81d18-openshift-default-66479b569c-nhgsq"]
Nov 03 20:26:26.735089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.735038    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz" podUID="6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" containerName="payload-processing" containerID="cri-o://6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940" gracePeriod=25
Nov 03 20:26:26.898697 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:26.898657    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:27.015788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.015761    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:26:27.082304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.082267    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-plugins-config-volume\") pod \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") "
Nov 03 20:26:27.082482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.082421    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-kube-api-access-npd2s\" (UniqueName: \"kubernetes.io/projected/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-kube-api-access-npd2s\") pod \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\" (UID: \"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4\") "
Nov 03 20:26:27.083659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.083463    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-plugins-config-volume" pod "6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" (UID: "6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:27.085036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.085010    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-kube-api-access-npd2s" pod "6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" (UID: "6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4"). InnerVolumeSpecName "kube-api-access-npd2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:27.184331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.184300    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:27.184331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.184333    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npd2s\" (UniqueName: \"kubernetes.io/projected/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4-kube-api-access-npd2s\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:27.439757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.439673    5282 generic.go:356] "Generic (PLEG): container finished" podID="6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" containerID="6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940" exitCode=0
Nov 03 20:26:27.439757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.439723    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"
Nov 03 20:26:27.439993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.439752    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz" event={"ID":"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4","Type":"ContainerDied","Data":"6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940"}
Nov 03 20:26:27.439993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.439804    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz" event={"ID":"6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4","Type":"ContainerDied","Data":"3eb2521966e956623b22cb1b945b779a58af245b10df25feeec82331577ab6df"}
Nov 03 20:26:27.439993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.439825    5282 scope.go:122] "RemoveContainer" containerID="6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940"
Nov 03 20:26:27.450076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.450064    5282 scope.go:122] "RemoveContainer" containerID="6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940"
Nov 03 20:26:27.450562 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:27.450529    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940\": container with ID starting with 6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940 not found: ID does not exist" containerID="6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940"
Nov 03 20:26:27.450634 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.450575    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940"} err="failed to get container status \"6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940\": rpc error: code = NotFound desc = could not find container \"6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940\": container with ID starting with 6982212696d8c85a56f98010a48435a2c2766b3c4ab90130338f2b4c52fe3940 not found: ID does not exist"
Nov 03 20:26:27.456816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.456793    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a030b91-7598-43b4-8554-ba348887cf30" path="/var/lib/kubelet/pods/5a030b91-7598-43b4-8554-ba348887cf30/volumes"
Nov 03 20:26:27.457363 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.457348    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" path="/var/lib/kubelet/pods/dfbfc761-e296-4114-b4e3-da2df1301934/volumes"
Nov 03 20:26:27.467427 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.467406    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"]
Nov 03 20:26:27.471820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.471795    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0ed729-d7bdd7f56-9cfnz"]
Nov 03 20:26:27.802561 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:27.802528    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:28.585331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:28.585296    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:29.105768 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:29.105723    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"]
Nov 03 20:26:29.165835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:29.165802    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:29.284829 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:29.284791    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:29.457509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:29.457423    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" path="/var/lib/kubelet/pods/6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4/volumes"
Nov 03 20:26:29.747077 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:29.747041    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:29.747505 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:29.747091    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:29.747505 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:29.747123    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:29.901914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:29.901881    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:30.119379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.119289    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:30.233722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.233682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:30.377889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.377749    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"]
Nov 03 20:26:30.378302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.378267    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" containerID="cri-o://338ee561b709de838fc6510528e23d2198b80d5ce138a8276c9e64d886505ef3" gracePeriod=30
Nov 03 20:26:30.381641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.381614    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"]
Nov 03 20:26:30.381876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.381842    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" containerID="cri-o://a04605cd504daad2a41c8a0dcc6b62e0a94f3e39507f1769aa89bb2966d19665" gracePeriod=30
Nov 03 20:26:30.387231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.387189    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"]
Nov 03 20:26:30.387433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.387410    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy" containerID="cri-o://df2198c0e2922a350431ae8b48ea1150fb619f8b8d2e5f1281070ed0ca95e4db" gracePeriod=30
Nov 03 20:26:30.429803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.429773    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:30.730270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.725662    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:30.870770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:30.870734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:31.024271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:31.024045    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:31.529449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:31.529409    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"]
Nov 03 20:26:31.634685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:31.633656    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:31.648343 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:31.648305    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:26:31.703807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:31.703771    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:31.704004 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:31.703832    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:31.752691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:31.752633    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:31.871510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:31.871412    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:32.517469 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.517439    5282 generic.go:356] "Generic (PLEG): container finished" podID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerID="72a6485d80472dea967a4cc228bffdd476cf5f446a5f34841be9fe5049bf8f60" exitCode=0
Nov 03 20:26:32.517608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.517519    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" event={"ID":"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac","Type":"ContainerDied","Data":"72a6485d80472dea967a4cc228bffdd476cf5f446a5f34841be9fe5049bf8f60"}
Nov 03 20:26:32.517663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.517629    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" event={"ID":"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac","Type":"ContainerDied","Data":"8ae9027a20271e6e7896f203633edaaef9622b2d117eb30c7625856a9c8f6f2c"}
Nov 03 20:26:32.517663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.517648    5282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ae9027a20271e6e7896f203633edaaef9622b2d117eb30c7625856a9c8f6f2c"
Nov 03 20:26:32.517749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.517732    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:26:32.645515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.645480    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-kube-api-access-8jx97\" (UniqueName: \"kubernetes.io/projected/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-kube-api-access-8jx97\") pod \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") "
Nov 03 20:26:32.645699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.645658    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-plugins-config-volume\") pod \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\" (UID: \"a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac\") "
Nov 03 20:26:32.646034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.646013    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-plugins-config-volume" pod "a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" (UID: "a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:32.646133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.646119    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:32.647727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.647700    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-kube-api-access-8jx97" pod "a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" (UID: "a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac"). InnerVolumeSpecName "kube-api-access-8jx97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:32.747838 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.747756    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jx97\" (UniqueName: \"kubernetes.io/projected/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac-kube-api-access-8jx97\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:32.967663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:32.967621    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:33.431997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.431954    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.149:50052\" within 1s: context deadline exceeded"
Nov 03 20:26:33.432158 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:26:33.432026    5282 logging.go:55] [core] [Channel #7723 SubChannel #7724]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.149:50052", ServerName: "10.133.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.149:50052: operation was canceled"
Nov 03 20:26:33.432158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.432035    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:26:33.523036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.523002    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"
Nov 03 20:26:33.541021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.540993    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"]
Nov 03 20:26:33.552059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.552027    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22a567-766dc74f8-4jdck"]
Nov 03 20:26:33.601157 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:33.601115    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:33.725506 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.725423    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:33.824219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.820083    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:33.824219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.820179    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:33.848267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.837231    5282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:33.848267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:33.837298    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:34.135564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:34.135529    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:34.850320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:34.850289    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:35.064116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.062382    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:35.247833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.247804    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:35.402521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.402490    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:35.455507 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.455480    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" path="/var/lib/kubelet/pods/a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac/volumes"
Nov 03 20:26:35.544005 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.543966    5282 generic.go:356] "Generic (PLEG): container finished" podID="8ebd8ea9-701c-4476-9380-9856f565e819" containerID="338ee561b709de838fc6510528e23d2198b80d5ce138a8276c9e64d886505ef3" exitCode=0
Nov 03 20:26:35.544260 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.544099    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" event={"ID":"8ebd8ea9-701c-4476-9380-9856f565e819","Type":"ContainerDied","Data":"338ee561b709de838fc6510528e23d2198b80d5ce138a8276c9e64d886505ef3"}
Nov 03 20:26:35.547322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.547105    5282 generic.go:356] "Generic (PLEG): container finished" podID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerID="a04605cd504daad2a41c8a0dcc6b62e0a94f3e39507f1769aa89bb2966d19665" exitCode=0
Nov 03 20:26:35.547322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.547227    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" event={"ID":"f0213ef7-a48f-4746-8c08-57b1a3961c1d","Type":"ContainerDied","Data":"a04605cd504daad2a41c8a0dcc6b62e0a94f3e39507f1769aa89bb2966d19665"}
Nov 03 20:26:35.549116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.549012    5282 generic.go:356] "Generic (PLEG): container finished" podID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerID="df2198c0e2922a350431ae8b48ea1150fb619f8b8d2e5f1281070ed0ca95e4db" exitCode=0
Nov 03 20:26:35.549116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.549040    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" event={"ID":"c08629be-9d3a-4e49-8935-e9f089de7ce9","Type":"ContainerDied","Data":"df2198c0e2922a350431ae8b48ea1150fb619f8b8d2e5f1281070ed0ca95e4db"}
Nov 03 20:26:35.658732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.658698    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:26:35.707898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.707865    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:35.727698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.727665    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:26:35.731570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.731547    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:26:35.796080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.795994    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-credential-socket\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.796080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796076    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-credential-socket\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.796379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796117    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-credential-socket\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.796379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796151    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-certs\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.796379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796243    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-kube-api-access-nxsfb\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-kube-api-access-nxsfb\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.796379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796273    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-istio-token\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-istio-token\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.796379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796308    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-socket\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.796379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796333    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-data\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.796379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796378    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istiod-ca-cert\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796429    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-envoy\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796436    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-credential-socket" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796456    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-credential-socket" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796501    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-certs" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796470    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-kube-api-access-bfxj4\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-kube-api-access-bfxj4\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796571    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-podinfo\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796643    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-socket\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796690    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-token\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-token\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.796751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796737    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-socket\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796786    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ebd8ea9-701c-4476-9380-9856f565e819-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebd8ea9-701c-4476-9380-9856f565e819-istio-podinfo\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796806    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istiod-ca-cert" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796822    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-certs\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796807    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-data" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796848    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-kube-api-access-4c2w5\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-kube-api-access-4c2w5\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796968    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-data\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.796995    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08629be-9d3a-4e49-8935-e9f089de7ce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08629be-9d3a-4e49-8935-e9f089de7ce9-istiod-ca-cert\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797038    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-podinfo\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797069    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ebd8ea9-701c-4476-9380-9856f565e819-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebd8ea9-701c-4476-9380-9856f565e819-istiod-ca-cert\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797090    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-socket" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797128    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-envoy\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797165    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-data\") pod \"c08629be-9d3a-4e49-8935-e9f089de7ce9\" (UID: \"c08629be-9d3a-4e49-8935-e9f089de7ce9\") "
Nov 03 20:26:35.797228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797214    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-token\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-token\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.797972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797266    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-envoy\") pod \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\" (UID: \"f0213ef7-a48f-4746-8c08-57b1a3961c1d\") "
Nov 03 20:26:35.797972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797310    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-certs\") pod \"8ebd8ea9-701c-4476-9380-9856f565e819\" (UID: \"8ebd8ea9-701c-4476-9380-9856f565e819\") "
Nov 03 20:26:35.797972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797708    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-credential-socket" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.797972 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.797917    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-certs" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.798207 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798091    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-data" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.798207 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798179    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798219    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798235    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798250    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798264    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798278    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798292    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798306    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.798702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798374    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08629be-9d3a-4e49-8935-e9f089de7ce9-istiod-ca-cert" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:35.798702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.798636    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-data" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.799408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.799379    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-kube-api-access-bfxj4" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "kube-api-access-bfxj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.799408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.799398    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-token" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.799592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.799471    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-socket" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.799660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.799617    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-istio-token" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.799721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.799661    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-socket" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.799832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.799783    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebd8ea9-701c-4476-9380-9856f565e819-istiod-ca-cert" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:35.799995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.799973    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-certs" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.801412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.801386    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-token" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.801958 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.801931    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-podinfo" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:35.802068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.802031    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-kube-api-access-4c2w5" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "kube-api-access-4c2w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.802147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.802084    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-kube-api-access-nxsfb" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "kube-api-access-nxsfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.802147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.802132    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-envoy" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.803210 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.803160    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-podinfo" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:35.803646 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.803603    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ebd8ea9-701c-4476-9380-9856f565e819-istio-podinfo" pod "8ebd8ea9-701c-4476-9380-9856f565e819" (UID: "8ebd8ea9-701c-4476-9380-9856f565e819"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:35.803775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.803756    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-envoy" pod "f0213ef7-a48f-4746-8c08-57b1a3961c1d" (UID: "f0213ef7-a48f-4746-8c08-57b1a3961c1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.803896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.803874    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-envoy" pod "c08629be-9d3a-4e49-8935-e9f089de7ce9" (UID: "c08629be-9d3a-4e49-8935-e9f089de7ce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:35.834122 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.834086    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"]
Nov 03 20:26:35.847783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.847750    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"]
Nov 03 20:26:35.899619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899589    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxsfb\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-kube-api-access-nxsfb\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899615    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899628    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899637    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfxj4\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-kube-api-access-bfxj4\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899648    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899657    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899666    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899674    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ebd8ea9-701c-4476-9380-9856f565e819-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899683    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ebd8ea9-701c-4476-9380-9856f565e819-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899691    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899700    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c2w5\" (UniqueName: \"kubernetes.io/projected/8ebd8ea9-701c-4476-9380-9856f565e819-kube-api-access-4c2w5\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899709    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899717    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08629be-9d3a-4e49-8935-e9f089de7ce9-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899726    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899735    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ebd8ea9-701c-4476-9380-9856f565e819-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899743    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899752    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08629be-9d3a-4e49-8935-e9f089de7ce9-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899760    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.899848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.899768    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0213ef7-a48f-4746-8c08-57b1a3961c1d-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.957823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:35.957676    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:36.089939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.089853    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:36.153541 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.153499    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:26:36.243339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.243309    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:36.359097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.359015    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:36.530388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.530343    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph" podUID="0ecdc4e5-5885-4c49-85b4-a58ed83fd584" containerName="payload-processing" containerID="cri-o://a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d" gracePeriod=25
Nov 03 20:26:36.557063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.557036    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"
Nov 03 20:26:36.557191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.557036    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2" event={"ID":"8ebd8ea9-701c-4476-9380-9856f565e819","Type":"ContainerDied","Data":"7cad4af3322bfb23b0105bdb1f70ca8c8aac14cb0e680cb2c049df65accaabd9"}
Nov 03 20:26:36.557290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.557188    5282 scope.go:122] "RemoveContainer" containerID="338ee561b709de838fc6510528e23d2198b80d5ce138a8276c9e64d886505ef3"
Nov 03 20:26:36.558678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.558651    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"
Nov 03 20:26:36.558678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.558656    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm" event={"ID":"f0213ef7-a48f-4746-8c08-57b1a3961c1d","Type":"ContainerDied","Data":"7400f9866268d6c9cbdced5c65fd43ade15b59976802bc05a272d2275f8baf73"}
Nov 03 20:26:36.560409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.560385    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp" event={"ID":"c08629be-9d3a-4e49-8935-e9f089de7ce9","Type":"ContainerDied","Data":"81672d88762961aa0a254302f1145ebb0eef9d6c9ac316d145b7b8f6e43fb120"}
Nov 03 20:26:36.560521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.560503    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"
Nov 03 20:26:36.583628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.583600    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:36.603621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.603579    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"]
Nov 03 20:26:36.607171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.607150    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-l5jk2"]
Nov 03 20:26:36.617271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.617219    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"]
Nov 03 20:26:36.622065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.622043    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-znjnm"]
Nov 03 20:26:36.632521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.632495    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"]
Nov 03 20:26:36.634784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.634761    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a476f8-openshift-default-8dc54c4fc-mglgp"]
Nov 03 20:26:36.644609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.644589    5282 scope.go:122] "RemoveContainer" containerID="a04605cd504daad2a41c8a0dcc6b62e0a94f3e39507f1769aa89bb2966d19665"
Nov 03 20:26:36.662727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.662700    5282 scope.go:122] "RemoveContainer" containerID="df2198c0e2922a350431ae8b48ea1150fb619f8b8d2e5f1281070ed0ca95e4db"
Nov 03 20:26:36.720830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.720760    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:36.802667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.802638    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:26:36.864191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.864153    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:36.915241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.915147    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-kube-api-access-22ktw\" (UniqueName: \"kubernetes.io/projected/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-kube-api-access-22ktw\") pod \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") "
Nov 03 20:26:36.915381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.915279    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-plugins-config-volume\") pod \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\" (UID: \"0ecdc4e5-5885-4c49-85b4-a58ed83fd584\") "
Nov 03 20:26:36.915647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.915621    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-plugins-config-volume" pod "0ecdc4e5-5885-4c49-85b4-a58ed83fd584" (UID: "0ecdc4e5-5885-4c49-85b4-a58ed83fd584"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:36.915839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.915824    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.917784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:36.917757    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-kube-api-access-22ktw" pod "0ecdc4e5-5885-4c49-85b4-a58ed83fd584" (UID: "0ecdc4e5-5885-4c49-85b4-a58ed83fd584"). InnerVolumeSpecName "kube-api-access-22ktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:37.017619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.017586    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ktw\" (UniqueName: \"kubernetes.io/projected/0ecdc4e5-5885-4c49-85b4-a58ed83fd584-kube-api-access-22ktw\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.151028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.150993    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:37.454654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.454610    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" path="/var/lib/kubelet/pods/8ebd8ea9-701c-4476-9380-9856f565e819/volumes"
Nov 03 20:26:37.455315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.455292    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" path="/var/lib/kubelet/pods/c08629be-9d3a-4e49-8935-e9f089de7ce9/volumes"
Nov 03 20:26:37.455805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.455785    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" path="/var/lib/kubelet/pods/f0213ef7-a48f-4746-8c08-57b1a3961c1d/volumes"
Nov 03 20:26:37.574626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.574592    5282 generic.go:356] "Generic (PLEG): container finished" podID="0ecdc4e5-5885-4c49-85b4-a58ed83fd584" containerID="a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d" exitCode=0
Nov 03 20:26:37.574840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.574739    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph" event={"ID":"0ecdc4e5-5885-4c49-85b4-a58ed83fd584","Type":"ContainerDied","Data":"a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d"}
Nov 03 20:26:37.574840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.574791    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph" event={"ID":"0ecdc4e5-5885-4c49-85b4-a58ed83fd584","Type":"ContainerDied","Data":"a68b8135d5ea6eac9a74180db074460dc16dc9645dcb015a2262ac06bd240367"}
Nov 03 20:26:37.574840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.574819    5282 scope.go:122] "RemoveContainer" containerID="a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d"
Nov 03 20:26:37.575019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.574926    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"
Nov 03 20:26:37.602480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.601918    5282 scope.go:122] "RemoveContainer" containerID="a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d"
Nov 03 20:26:37.602671 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:37.602477    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d\": container with ID starting with a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d not found: ID does not exist" containerID="a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d"
Nov 03 20:26:37.602671 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.602516    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d"} err="failed to get container status \"a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d\": rpc error: code = NotFound desc = could not find container \"a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d\": container with ID starting with a9d7601549dcc38e785046ebe3081b77f57c2b1b8b9b552e5c113332f0df237d not found: ID does not exist"
Nov 03 20:26:37.606052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.606018    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"]
Nov 03 20:26:37.611812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.611776    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-944e83-795689f8b-t6tph"]
Nov 03 20:26:37.972991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:37.972955    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:38.783870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:38.783839    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:39.454769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:39.454733    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecdc4e5-5885-4c49-85b4-a58ed83fd584" path="/var/lib/kubelet/pods/0ecdc4e5-5885-4c49-85b4-a58ed83fd584/volumes"
Nov 03 20:26:39.552560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:39.552465    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:40.260949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:40.260902    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:40.894060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:40.894026    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:41.260010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:41.259976    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:41.386696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:41.386652    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:41.810707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:41.810670    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:42.253930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:42.253897    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:42.384160 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:42.384128    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:42.524701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:42.524614    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"]
Nov 03 20:26:42.525180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:42.524849    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy" containerID="cri-o://f371e9464e50f7ea5a5bfc05ae38d2e0fce4c3fd5514dcdf15bc3d8c34d1a48f" gracePeriod=30
Nov 03 20:26:42.527014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:42.526981    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:42.768107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:42.768062    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:43.078862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:43.078825    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:43.599235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:43.596935    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:43.669826 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:43.669795    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:44.117889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.117851    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:44.129604 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.129563    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z" podUID="76eaf966-caf0-4394-9f02-488f93755253" containerName="main" containerID="cri-o://a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb" gracePeriod=45
Nov 03 20:26:44.137784 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.137750    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"]
Nov 03 20:26:44.165346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.165315    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"]
Nov 03 20:26:44.287826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.287778    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:44.420729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.420706    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:26:44.445504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.445476    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:44.525037 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525000    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76eaf966-caf0-4394-9f02-488f93755253-tls-certs\" (UniqueName: \"kubernetes.io/secret/76eaf966-caf0-4394-9f02-488f93755253-tls-certs\") pod \"76eaf966-caf0-4394-9f02-488f93755253\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") "
Nov 03 20:26:44.525277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525050    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-model-cache\") pod \"76eaf966-caf0-4394-9f02-488f93755253\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") "
Nov 03 20:26:44.525277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525080    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-home\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-home\") pod \"76eaf966-caf0-4394-9f02-488f93755253\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") "
Nov 03 20:26:44.525277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525156    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eaf966-caf0-4394-9f02-488f93755253-kube-api-access-mq7hn\" (UniqueName: \"kubernetes.io/projected/76eaf966-caf0-4394-9f02-488f93755253-kube-api-access-mq7hn\") pod \"76eaf966-caf0-4394-9f02-488f93755253\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") "
Nov 03 20:26:44.525277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525238    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-tmp-dir\") pod \"76eaf966-caf0-4394-9f02-488f93755253\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") "
Nov 03 20:26:44.525522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525354    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-dshm\") pod \"76eaf966-caf0-4394-9f02-488f93755253\" (UID: \"76eaf966-caf0-4394-9f02-488f93755253\") "
Nov 03 20:26:44.525522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525393    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-model-cache" pod "76eaf966-caf0-4394-9f02-488f93755253" (UID: "76eaf966-caf0-4394-9f02-488f93755253"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:44.525522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525401    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-home" pod "76eaf966-caf0-4394-9f02-488f93755253" (UID: "76eaf966-caf0-4394-9f02-488f93755253"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:44.525628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525582    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-tmp-dir" pod "76eaf966-caf0-4394-9f02-488f93755253" (UID: "76eaf966-caf0-4394-9f02-488f93755253"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:44.525948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525921    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:44.526058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525952    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:44.526058 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.525968    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:44.527564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.527538    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-dshm" pod "76eaf966-caf0-4394-9f02-488f93755253" (UID: "76eaf966-caf0-4394-9f02-488f93755253"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:44.527659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.527568    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76eaf966-caf0-4394-9f02-488f93755253-tls-certs" pod "76eaf966-caf0-4394-9f02-488f93755253" (UID: "76eaf966-caf0-4394-9f02-488f93755253"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:44.527659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.527571    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eaf966-caf0-4394-9f02-488f93755253-kube-api-access-mq7hn" pod "76eaf966-caf0-4394-9f02-488f93755253" (UID: "76eaf966-caf0-4394-9f02-488f93755253"). InnerVolumeSpecName "kube-api-access-mq7hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:44.674707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.674616    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eaf966-caf0-4394-9f02-488f93755253-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:44.674707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.674653    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76eaf966-caf0-4394-9f02-488f93755253-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:44.674707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.674671    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7hn\" (UniqueName: \"kubernetes.io/projected/76eaf966-caf0-4394-9f02-488f93755253-kube-api-access-mq7hn\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:44.679012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.675733    5282 generic.go:356] "Generic (PLEG): container finished" podID="76eaf966-caf0-4394-9f02-488f93755253" containerID="a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb" exitCode=0
Nov 03 20:26:44.679012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.675969    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z" event={"ID":"76eaf966-caf0-4394-9f02-488f93755253","Type":"ContainerDied","Data":"a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb"}
Nov 03 20:26:44.679012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.676010    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z" event={"ID":"76eaf966-caf0-4394-9f02-488f93755253","Type":"ContainerDied","Data":"36a399446b4bb7df90f409ac4b545fa87590ad7a5f32dc05ce169851cf96f825"}
Nov 03 20:26:44.679012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.676031    5282 scope.go:122] "RemoveContainer" containerID="a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb"
Nov 03 20:26:44.679012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.676164    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"
Nov 03 20:26:44.711013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.708994    5282 scope.go:122] "RemoveContainer" containerID="a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb"
Nov 03 20:26:44.711013 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:44.709495    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb\": container with ID starting with a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb not found: ID does not exist" containerID="a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb"
Nov 03 20:26:44.711013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.709536    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb"} err="failed to get container status \"a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb\": rpc error: code = NotFound desc = could not find container \"a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb\": container with ID starting with a5ae6e94fd8c30ee166838448a8162588574a22360819f5a3939cf4bcb74a7cb not found: ID does not exist"
Nov 03 20:26:44.717368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.717334    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"]
Nov 03 20:26:44.717515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.717392    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-456f55/auth-test-model-456f55-kserve-68dbcfc989-hr96z"]
Nov 03 20:26:44.741849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.741818    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:44.749377 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:44.749344    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:44.875608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:44.875575    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:45.040238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:45.040177    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"]
Nov 03 20:26:45.047643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:45.040485    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" containerID="cri-o://185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd" gracePeriod=30
Nov 03 20:26:45.208216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:45.194175    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:45.456764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:45.455874    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76eaf966-caf0-4394-9f02-488f93755253" path="/var/lib/kubelet/pods/76eaf966-caf0-4394-9f02-488f93755253/volumes"
Nov 03 20:26:46.067904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:46.067875    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:46.101899 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:46.101853    5282 patch_prober.go:29] interesting pod/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:46.102188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:46.102161    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:46.810027 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:46.809992    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:47.435703 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.435668    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:47.607516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.607474    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:47.697059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.696982    5282 generic.go:356] "Generic (PLEG): container finished" podID="347cb059-324b-4249-b74b-2a18fd8d045a" containerID="f371e9464e50f7ea5a5bfc05ae38d2e0fce4c3fd5514dcdf15bc3d8c34d1a48f" exitCode=0
Nov 03 20:26:47.697221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.697055    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" event={"ID":"347cb059-324b-4249-b74b-2a18fd8d045a","Type":"ContainerDied","Data":"f371e9464e50f7ea5a5bfc05ae38d2e0fce4c3fd5514dcdf15bc3d8c34d1a48f"}
Nov 03 20:26:47.697221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.697118    5282 scope.go:122] "RemoveContainer" containerID="5bcedfd738e5b4906ce66fa796b322c0b81482e249ec88e229de9c87bca2b1ec"
Nov 03 20:26:47.762222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.762158    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:47.803947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.803923    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:26:47.820805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.820774    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-kube-api-access-lg6t2\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-kube-api-access-lg6t2\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.820983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.820845    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/347cb059-324b-4249-b74b-2a18fd8d045a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347cb059-324b-4249-b74b-2a18fd8d045a-istio-podinfo\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821007    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-data\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821178 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821154    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347cb059-324b-4249-b74b-2a18fd8d045a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347cb059-324b-4249-b74b-2a18fd8d045a-istiod-ca-cert\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821221    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-istio-token\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-istio-token\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821291    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-envoy\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821298    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-data" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:47.821392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821338    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-credential-socket\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821365    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-socket\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821396    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-certs\") pod \"347cb059-324b-4249-b74b-2a18fd8d045a\" (UID: \"347cb059-324b-4249-b74b-2a18fd8d045a\") "
Nov 03 20:26:47.821710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821597    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-credential-socket" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:47.821710 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821524    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347cb059-324b-4249-b74b-2a18fd8d045a-istiod-ca-cert" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:47.821908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821865    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-socket" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:47.822025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821956    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.822025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821978    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/347cb059-324b-4249-b74b-2a18fd8d045a-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.822025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.821993    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.822536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.822509    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-certs" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:47.823523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.823493    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/347cb059-324b-4249-b74b-2a18fd8d045a-istio-podinfo" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:47.823603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.823528    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-kube-api-access-lg6t2" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "kube-api-access-lg6t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:47.823603 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.823548    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-istio-token" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:47.823788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.823772    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-envoy" pod "347cb059-324b-4249-b74b-2a18fd8d045a" (UID: "347cb059-324b-4249-b74b-2a18fd8d045a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:47.881723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.881661    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:47.925545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.924136    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/347cb059-324b-4249-b74b-2a18fd8d045a-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.925545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.924177    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.925545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.924216    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.925545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.924236    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.925545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.924268    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/347cb059-324b-4249-b74b-2a18fd8d045a-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.925545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.924282    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg6t2\" (UniqueName: \"kubernetes.io/projected/347cb059-324b-4249-b74b-2a18fd8d045a-kube-api-access-lg6t2\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:47.999235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:47.999188    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"]
Nov 03 20:26:48.017619 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.017587    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"]
Nov 03 20:26:48.124624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.124576    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:48.247136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.247103    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:48.704069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.704031    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"
Nov 03 20:26:48.704629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.704076    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl" event={"ID":"347cb059-324b-4249-b74b-2a18fd8d045a","Type":"ContainerDied","Data":"5607d30570879492306a97b565a5a174f08af8a8f238a65ae882d81ee6168835"}
Nov 03 20:26:48.704629 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.704138    5282 scope.go:122] "RemoveContainer" containerID="f371e9464e50f7ea5a5bfc05ae38d2e0fce4c3fd5514dcdf15bc3d8c34d1a48f"
Nov 03 20:26:48.709705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.709653    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:48.739721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.739681    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"]
Nov 03 20:26:48.754593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:48.754548    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-27d3ad-openshift-default-85d7dd4c5-98tcl"]
Nov 03 20:26:49.139223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.139115    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg" podUID="5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" containerName="payload-processing" containerID="cri-o://45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7" gracePeriod=25
Nov 03 20:26:49.166435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.166406    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg" podUID="529c2f87-65fc-44d1-94dd-c9f52ff20342" containerName="payload-pre-processing" containerID="cri-o://292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747" gracePeriod=25
Nov 03 20:26:49.435385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.435358    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:26:49.457140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.456649    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-plugins-config-volume\") pod \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") "
Nov 03 20:26:49.457140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.457096    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-kube-api-access-qpf5w\" (UniqueName: \"kubernetes.io/projected/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-kube-api-access-qpf5w\") pod \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\" (UID: \"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8\") "
Nov 03 20:26:49.457416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.457004    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-plugins-config-volume" pod "5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" (UID: "5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:49.458113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.457778    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.458895 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.458866    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" path="/var/lib/kubelet/pods/347cb059-324b-4249-b74b-2a18fd8d045a/volumes"
Nov 03 20:26:49.461386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.461360    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-kube-api-access-qpf5w" pod "5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" (UID: "5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8"). InnerVolumeSpecName "kube-api-access-qpf5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:49.464273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.464252    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:26:49.502627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.502584    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:49.558255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.558221    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529c2f87-65fc-44d1-94dd-c9f52ff20342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529c2f87-65fc-44d1-94dd-c9f52ff20342-plugins-config-volume\") pod \"529c2f87-65fc-44d1-94dd-c9f52ff20342\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") "
Nov 03 20:26:49.558459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.558399    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529c2f87-65fc-44d1-94dd-c9f52ff20342-kube-api-access-s9fnz\" (UniqueName: \"kubernetes.io/projected/529c2f87-65fc-44d1-94dd-c9f52ff20342-kube-api-access-s9fnz\") pod \"529c2f87-65fc-44d1-94dd-c9f52ff20342\" (UID: \"529c2f87-65fc-44d1-94dd-c9f52ff20342\") "
Nov 03 20:26:49.558661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.558637    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529c2f87-65fc-44d1-94dd-c9f52ff20342-plugins-config-volume" pod "529c2f87-65fc-44d1-94dd-c9f52ff20342" (UID: "529c2f87-65fc-44d1-94dd-c9f52ff20342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:49.558805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.558787    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529c2f87-65fc-44d1-94dd-c9f52ff20342-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.558854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.558813    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpf5w\" (UniqueName: \"kubernetes.io/projected/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8-kube-api-access-qpf5w\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.560523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.560502    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529c2f87-65fc-44d1-94dd-c9f52ff20342-kube-api-access-s9fnz" pod "529c2f87-65fc-44d1-94dd-c9f52ff20342" (UID: "529c2f87-65fc-44d1-94dd-c9f52ff20342"). InnerVolumeSpecName "kube-api-access-s9fnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:49.659483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.659411    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9fnz\" (UniqueName: \"kubernetes.io/projected/529c2f87-65fc-44d1-94dd-c9f52ff20342-kube-api-access-s9fnz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:49.712310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.712270    5282 generic.go:356] "Generic (PLEG): container finished" podID="529c2f87-65fc-44d1-94dd-c9f52ff20342" containerID="292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747" exitCode=0
Nov 03 20:26:49.712928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.712344    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"
Nov 03 20:26:49.712928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.712382    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg" event={"ID":"529c2f87-65fc-44d1-94dd-c9f52ff20342","Type":"ContainerDied","Data":"292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747"}
Nov 03 20:26:49.712928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.712487    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg" event={"ID":"529c2f87-65fc-44d1-94dd-c9f52ff20342","Type":"ContainerDied","Data":"3774d3f7bcdf28a871a1436d1136ed97a1a8995f05acd5376e8bb62eb081da6a"}
Nov 03 20:26:49.712928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.712516    5282 scope.go:122] "RemoveContainer" containerID="292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747"
Nov 03 20:26:49.718690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.718662    5282 generic.go:356] "Generic (PLEG): container finished" podID="5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" containerID="45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7" exitCode=0
Nov 03 20:26:49.718836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.718709    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg" event={"ID":"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8","Type":"ContainerDied","Data":"45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7"}
Nov 03 20:26:49.718836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.718736    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"
Nov 03 20:26:49.718836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.718753    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg" event={"ID":"5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8","Type":"ContainerDied","Data":"2cad1177ce450727adcef130dab98c2de93c4c904e336677bdc0aec22a3929f6"}
Nov 03 20:26:49.726395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.726373    5282 scope.go:122] "RemoveContainer" containerID="292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747"
Nov 03 20:26:49.726685 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:49.726655    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747\": container with ID starting with 292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747 not found: ID does not exist" containerID="292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747"
Nov 03 20:26:49.726802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.726696    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747"} err="failed to get container status \"292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747\": rpc error: code = NotFound desc = could not find container \"292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747\": container with ID starting with 292da411197ba6ed9909fa2ed84a358385c2247c251409295d226bfb6838a747 not found: ID does not exist"
Nov 03 20:26:49.726802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.726721    5282 scope.go:122] "RemoveContainer" containerID="45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7"
Nov 03 20:26:49.738093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.738072    5282 scope.go:122] "RemoveContainer" containerID="45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7"
Nov 03 20:26:49.738477 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:49.738447    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7\": container with ID starting with 45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7 not found: ID does not exist" containerID="45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7"
Nov 03 20:26:49.738596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.738487    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7"} err="failed to get container status \"45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7\": rpc error: code = NotFound desc = could not find container \"45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7\": container with ID starting with 45120b95f4746713f3063600247044462ae25dd4b0378d23510a1bcfc7410cc7 not found: ID does not exist"
Nov 03 20:26:49.739475 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.739453    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"]
Nov 03 20:26:49.746281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.746256    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-456f55-5898c984ff-9w9bg"]
Nov 03 20:26:49.758082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.758059    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"]
Nov 03 20:26:49.763774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.763756    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-456f55-76868f5c98-z9kwg"]
Nov 03 20:26:49.776811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.776789    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"]
Nov 03 20:26:49.783803 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.783784    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"]
Nov 03 20:26:49.911599 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:49.911456    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:50.309038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.309013    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:26:50.365998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.365968    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-certs\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366020    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istiod-ca-cert\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366074    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-credential-socket\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366377    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-podinfo\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366428    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-data\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366472    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-socket\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366500    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-token\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-token\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366571    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-envoy\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366598    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-kube-api-access-tw65x\") pod \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\" (UID: \"dc6e5cd9-7b07-4b92-881b-81deaf1524c2\") "
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366467    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-credential-socket" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366545    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istiod-ca-cert" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366654    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-data" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.366937 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.366894    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-certs" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.367597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.367219    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-socket" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.367597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.367253    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.367597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.367276    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.367597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.367291    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.367597 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.367305    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.369092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.369069    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-token" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:50.369441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.369416    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-envoy" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:50.369529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.369426    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-kube-api-access-tw65x" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "kube-api-access-tw65x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:50.369529 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.369500    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-podinfo" pod "dc6e5cd9-7b07-4b92-881b-81deaf1524c2" (UID: "dc6e5cd9-7b07-4b92-881b-81deaf1524c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:50.396414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.396382    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:50.468577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.468492    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.468577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.468519    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw65x\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-kube-api-access-tw65x\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.468577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.468529    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.468577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.468539    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.468577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.468558    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6e5cd9-7b07-4b92-881b-81deaf1524c2-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:50.727835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.727804    5282 generic.go:356] "Generic (PLEG): container finished" podID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerID="185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd" exitCode=0
Nov 03 20:26:50.728366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.727855    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" event={"ID":"dc6e5cd9-7b07-4b92-881b-81deaf1524c2","Type":"ContainerDied","Data":"185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd"}
Nov 03 20:26:50.728366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.727859    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"
Nov 03 20:26:50.728366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.727894    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv" event={"ID":"dc6e5cd9-7b07-4b92-881b-81deaf1524c2","Type":"ContainerDied","Data":"d86635013f34c2704e5f7593d272b3d51ef977cf012002d5220e1e80a7c74117"}
Nov 03 20:26:50.728366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.727915    5282 scope.go:122] "RemoveContainer" containerID="185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd"
Nov 03 20:26:50.738574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.738552    5282 scope.go:122] "RemoveContainer" containerID="185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd"
Nov 03 20:26:50.738838 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:50.738820    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd\": container with ID starting with 185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd not found: ID does not exist" containerID="185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd"
Nov 03 20:26:50.738888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.738845    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd"} err="failed to get container status \"185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd\": rpc error: code = NotFound desc = could not find container \"185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd\": container with ID starting with 185d337576bfa8a08843e429b0d273ffbbcdd1d11460ac0c04c1324d765265fd not found: ID does not exist"
Nov 03 20:26:50.752931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.752904    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"]
Nov 03 20:26:50.756066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.756046    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0ed729-openshift-default-5b668fd6bc-8mrhv"]
Nov 03 20:26:50.855542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.855508    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" containerID="cri-o://9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6" gracePeriod=45
Nov 03 20:26:50.868667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:50.868569    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main" containerID="cri-o://989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b" gracePeriod=45
Nov 03 20:26:51.085214 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.084888    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:51.193782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.193752    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:26:51.251565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.251486    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:26:51.278652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.278615    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68d5d678-8552-4d54-a140-210d27813c3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/68d5d678-8552-4d54-a140-210d27813c3d-tls-certs\") pod \"68d5d678-8552-4d54-a140-210d27813c3d\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") "
Nov 03 20:26:51.278843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.278713    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d5d678-8552-4d54-a140-210d27813c3d-kube-api-access-l8fmc\" (UniqueName: \"kubernetes.io/projected/68d5d678-8552-4d54-a140-210d27813c3d-kube-api-access-l8fmc\") pod \"68d5d678-8552-4d54-a140-210d27813c3d\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") "
Nov 03 20:26:51.278843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.278788    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tls-certs\") pod \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") "
Nov 03 20:26:51.279491 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.279448    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-tmp-dir" pod "68d5d678-8552-4d54-a140-210d27813c3d" (UID: "68d5d678-8552-4d54-a140-210d27813c3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.279656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.279637    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-tmp-dir\") pod \"68d5d678-8552-4d54-a140-210d27813c3d\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") "
Nov 03 20:26:51.279884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.279794    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-dshm\") pod \"68d5d678-8552-4d54-a140-210d27813c3d\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") "
Nov 03 20:26:51.279983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.279921    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f873dc7e-faaa-4b5d-8b32-298726b5aef8-kube-api-access-2zj22\" (UniqueName: \"kubernetes.io/projected/f873dc7e-faaa-4b5d-8b32-298726b5aef8-kube-api-access-2zj22\") pod \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") "
Nov 03 20:26:51.279983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.279974    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-home\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-home\") pod \"68d5d678-8552-4d54-a140-210d27813c3d\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") "
Nov 03 20:26:51.280098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.280027    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-model-cache\") pod \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") "
Nov 03 20:26:51.280098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.280066    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-model-cache\") pod \"68d5d678-8552-4d54-a140-210d27813c3d\" (UID: \"68d5d678-8552-4d54-a140-210d27813c3d\") "
Nov 03 20:26:51.280261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.280165    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-dshm\") pod \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") "
Nov 03 20:26:51.280261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.280246    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-home\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-home\") pod \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") "
Nov 03 20:26:51.280385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.280291    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tmp-dir\") pod \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\" (UID: \"f873dc7e-faaa-4b5d-8b32-298726b5aef8\") "
Nov 03 20:26:51.280736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.280687    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-home" pod "68d5d678-8552-4d54-a140-210d27813c3d" (UID: "68d5d678-8552-4d54-a140-210d27813c3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.280978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.280952    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-model-cache" pod "f873dc7e-faaa-4b5d-8b32-298726b5aef8" (UID: "f873dc7e-faaa-4b5d-8b32-298726b5aef8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.281353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.281229    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-home" pod "f873dc7e-faaa-4b5d-8b32-298726b5aef8" (UID: "f873dc7e-faaa-4b5d-8b32-298726b5aef8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.281514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.281474    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tmp-dir" pod "f873dc7e-faaa-4b5d-8b32-298726b5aef8" (UID: "f873dc7e-faaa-4b5d-8b32-298726b5aef8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.281591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.281494    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-model-cache" pod "68d5d678-8552-4d54-a140-210d27813c3d" (UID: "68d5d678-8552-4d54-a140-210d27813c3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.281648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.281614    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.281648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.281636    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.281763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.281650    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.281763 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.281664    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.283892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.283845    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68d5d678-8552-4d54-a140-210d27813c3d-tls-certs" pod "68d5d678-8552-4d54-a140-210d27813c3d" (UID: "68d5d678-8552-4d54-a140-210d27813c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:51.283892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.283866    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tls-certs" pod "f873dc7e-faaa-4b5d-8b32-298726b5aef8" (UID: "f873dc7e-faaa-4b5d-8b32-298726b5aef8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:51.283892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.283854    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f873dc7e-faaa-4b5d-8b32-298726b5aef8-kube-api-access-2zj22" pod "f873dc7e-faaa-4b5d-8b32-298726b5aef8" (UID: "f873dc7e-faaa-4b5d-8b32-298726b5aef8"). InnerVolumeSpecName "kube-api-access-2zj22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:51.284162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.283890    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d5d678-8552-4d54-a140-210d27813c3d-kube-api-access-l8fmc" pod "68d5d678-8552-4d54-a140-210d27813c3d" (UID: "68d5d678-8552-4d54-a140-210d27813c3d"). InnerVolumeSpecName "kube-api-access-l8fmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:51.284446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.284419    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-dshm" pod "f873dc7e-faaa-4b5d-8b32-298726b5aef8" (UID: "f873dc7e-faaa-4b5d-8b32-298726b5aef8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.285645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.285625    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-dshm" pod "68d5d678-8552-4d54-a140-210d27813c3d" (UID: "68d5d678-8552-4d54-a140-210d27813c3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:51.382687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382655    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.382687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382687    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zj22\" (UniqueName: \"kubernetes.io/projected/f873dc7e-faaa-4b5d-8b32-298726b5aef8-kube-api-access-2zj22\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.382897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382700    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d5d678-8552-4d54-a140-210d27813c3d-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.382897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382710    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.382897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382719    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.382897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382727    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d5d678-8552-4d54-a140-210d27813c3d-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.382897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382735    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8fmc\" (UniqueName: \"kubernetes.io/projected/68d5d678-8552-4d54-a140-210d27813c3d-kube-api-access-l8fmc\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.382897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.382743    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873dc7e-faaa-4b5d-8b32-298726b5aef8-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:51.459818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.459783    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529c2f87-65fc-44d1-94dd-c9f52ff20342" path="/var/lib/kubelet/pods/529c2f87-65fc-44d1-94dd-c9f52ff20342/volumes"
Nov 03 20:26:51.460233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.460187    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" path="/var/lib/kubelet/pods/5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8/volumes"
Nov 03 20:26:51.460657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.460644    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" path="/var/lib/kubelet/pods/dc6e5cd9-7b07-4b92-881b-81deaf1524c2/volumes"
Nov 03 20:26:51.735383 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.735348    5282 generic.go:356] "Generic (PLEG): container finished" podID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerID="989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b" exitCode=0
Nov 03 20:26:51.735844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.735407    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"
Nov 03 20:26:51.735844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.735437    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" event={"ID":"f873dc7e-faaa-4b5d-8b32-298726b5aef8","Type":"ContainerDied","Data":"989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b"}
Nov 03 20:26:51.735844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.735485    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf" event={"ID":"f873dc7e-faaa-4b5d-8b32-298726b5aef8","Type":"ContainerDied","Data":"86fc2258952368049472257fd559021470efbbcc97668c9c71fca548edae330a"}
Nov 03 20:26:51.735844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.735507    5282 scope.go:122] "RemoveContainer" containerID="989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b"
Nov 03 20:26:51.736971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.736951    5282 generic.go:356] "Generic (PLEG): container finished" podID="68d5d678-8552-4d54-a140-210d27813c3d" containerID="9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6" exitCode=0
Nov 03 20:26:51.737076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.737056    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"
Nov 03 20:26:51.737158 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.737054    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" event={"ID":"68d5d678-8552-4d54-a140-210d27813c3d","Type":"ContainerDied","Data":"9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6"}
Nov 03 20:26:51.737300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.737278    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" event={"ID":"68d5d678-8552-4d54-a140-210d27813c3d","Type":"ContainerDied","Data":"38ff8ce6b35e19b910033e87610d0e1a88dc20743f347e543bb6c58beefb37fe"}
Nov 03 20:26:51.746038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.746016    5282 scope.go:122] "RemoveContainer" containerID="989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b"
Nov 03 20:26:51.746471 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:51.746445    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b\": container with ID starting with 989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b not found: ID does not exist" containerID="989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b"
Nov 03 20:26:51.746570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.746478    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b"} err="failed to get container status \"989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b\": rpc error: code = NotFound desc = could not find container \"989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b\": container with ID starting with 989c4dd28a4a1cb920634650bfeb2de4fa7cb97510c06716ab461f2576998f2b not found: ID does not exist"
Nov 03 20:26:51.746570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.746494    5282 scope.go:122] "RemoveContainer" containerID="9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6"
Nov 03 20:26:51.755939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.755904    5282 scope.go:122] "RemoveContainer" containerID="9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6"
Nov 03 20:26:51.756273 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:51.756246    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6\": container with ID starting with 9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6 not found: ID does not exist" containerID="9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6"
Nov 03 20:26:51.756359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.756288    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6"} err="failed to get container status \"9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6\": rpc error: code = NotFound desc = could not find container \"9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6\": container with ID starting with 9c4286477637a42b139369a6824d283faf6ba862cbbedc87a11bb86a253ac7b6 not found: ID does not exist"
Nov 03 20:26:51.758062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.758039    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"]
Nov 03 20:26:51.761164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.761141    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/simulator-w0-120fe3-kserve-f5bbcfc96-lhhpf"]
Nov 03 20:26:51.771596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.771573    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"]
Nov 03 20:26:51.775496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.775469    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95"]
Nov 03 20:26:51.803072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:51.803045    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:52.154498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:52.154415    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-120fe3/premium-w0-120fe3-kserve-6d54b95c8f-9jh95" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:52.583795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:52.583749    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:52.717001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:52.716968    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:52.836643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:52.836557    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:52.952220 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:52.951749    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:53.422352 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:53.422294    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:53.446592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:53.446551    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"]
Nov 03 20:26:53.457364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:53.457332    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d5d678-8552-4d54-a140-210d27813c3d" path="/var/lib/kubelet/pods/68d5d678-8552-4d54-a140-210d27813c3d/volumes"
Nov 03 20:26:53.457994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:53.457970    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" path="/var/lib/kubelet/pods/f873dc7e-faaa-4b5d-8b32-298726b5aef8/volumes"
Nov 03 20:26:53.557076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:53.557042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:53.803825 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:53.803791    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:54.485822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:54.485778    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:54.657904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:54.657871    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:55.181896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:55.181866    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:55.393222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:55.391580    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:55.472971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:55.472857    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"]
Nov 03 20:26:55.473151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:55.473109    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy" containerID="cri-o://f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e" gracePeriod=30
Nov 03 20:26:55.534819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:55.534785    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:55.658007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:55.657977    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:56.448978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:56.448942    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:57.561639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:57.561604    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:58.447100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:58.447056    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" podUID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" containerName="payload-pre-processing" containerID="cri-o://fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501" gracePeriod=25
Nov 03 20:26:58.465479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:58.465455    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:59.382229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:59.382180    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:59.786057 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:26:59.786020    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:59.827249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:59.827215    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"]
Nov 03 20:26:59.827495 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:59.827444    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh" podUID="777432e5-8d85-41bc-b1d3-851ee8926850" containerName="maas-api" containerID="cri-o://4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467" gracePeriod=30
Nov 03 20:26:59.898757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:59.898726    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:26:59.985771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:26:59.985738    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:27:00.113138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.113018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:00.160590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.160564    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:27:00.248191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.248153    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777432e5-8d85-41bc-b1d3-851ee8926850-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777432e5-8d85-41bc-b1d3-851ee8926850-openshift-service-ca\") pod \"777432e5-8d85-41bc-b1d3-851ee8926850\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") "
Nov 03 20:27:00.248394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.248261    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-metrics-certs\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-metrics-certs\") pod \"777432e5-8d85-41bc-b1d3-851ee8926850\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") "
Nov 03 20:27:00.248394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.248319    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777432e5-8d85-41bc-b1d3-851ee8926850-kube-api-access-jt2zz\" (UniqueName: \"kubernetes.io/projected/777432e5-8d85-41bc-b1d3-851ee8926850-kube-api-access-jt2zz\") pod \"777432e5-8d85-41bc-b1d3-851ee8926850\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") "
Nov 03 20:27:00.248527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.248469    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777432e5-8d85-41bc-b1d3-851ee8926850-openshift-service-ca" pod "777432e5-8d85-41bc-b1d3-851ee8926850" (UID: "777432e5-8d85-41bc-b1d3-851ee8926850"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:00.248582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.248526    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls\") pod \"777432e5-8d85-41bc-b1d3-851ee8926850\" (UID: \"777432e5-8d85-41bc-b1d3-851ee8926850\") "
Nov 03 20:27:00.249079 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.249048    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777432e5-8d85-41bc-b1d3-851ee8926850-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.250665 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.250635    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777432e5-8d85-41bc-b1d3-851ee8926850-kube-api-access-jt2zz" pod "777432e5-8d85-41bc-b1d3-851ee8926850" (UID: "777432e5-8d85-41bc-b1d3-851ee8926850"). InnerVolumeSpecName "kube-api-access-jt2zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:00.250781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.250682    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-metrics-certs" pod "777432e5-8d85-41bc-b1d3-851ee8926850" (UID: "777432e5-8d85-41bc-b1d3-851ee8926850"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:00.250852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.250776    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls" pod "777432e5-8d85-41bc-b1d3-851ee8926850" (UID: "777432e5-8d85-41bc-b1d3-851ee8926850"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:00.350306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.350267    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.350306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.350298    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2zz\" (UniqueName: \"kubernetes.io/projected/777432e5-8d85-41bc-b1d3-851ee8926850-kube-api-access-jt2zz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.350306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.350309    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777432e5-8d85-41bc-b1d3-851ee8926850-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.440287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.431971    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:00.671224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.667473    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:00.781930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.781906    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:27:00.807274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.807169    5282 generic.go:356] "Generic (PLEG): container finished" podID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerID="f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e" exitCode=0
Nov 03 20:27:00.807274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.807225    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"
Nov 03 20:27:00.807274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.807254    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" event={"ID":"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3","Type":"ContainerDied","Data":"f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e"}
Nov 03 20:27:00.807524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.807338    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x" event={"ID":"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3","Type":"ContainerDied","Data":"05a59703c5b9202948a8b997a6e442465a909c73fa5f5e3cb546287f3fc02193"}
Nov 03 20:27:00.807524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.807392    5282 scope.go:122] "RemoveContainer" containerID="f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e"
Nov 03 20:27:00.815990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.815849    5282 generic.go:356] "Generic (PLEG): container finished" podID="777432e5-8d85-41bc-b1d3-851ee8926850" containerID="4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467" exitCode=0
Nov 03 20:27:00.816116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.816062    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh" event={"ID":"777432e5-8d85-41bc-b1d3-851ee8926850","Type":"ContainerDied","Data":"4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467"}
Nov 03 20:27:00.816209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.816112    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh" event={"ID":"777432e5-8d85-41bc-b1d3-851ee8926850","Type":"ContainerDied","Data":"1f0371a7d11ebb9245afc7410953a379badf2b905f2488e690a0d38caa0783ed"}
Nov 03 20:27:00.816209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.816186    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"
Nov 03 20:27:00.824693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.824672    5282 scope.go:122] "RemoveContainer" containerID="f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e"
Nov 03 20:27:00.825140 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:00.825003    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e\": container with ID starting with f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e not found: ID does not exist" containerID="f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e"
Nov 03 20:27:00.825140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.825039    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e"} err="failed to get container status \"f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e\": rpc error: code = NotFound desc = could not find container \"f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e\": container with ID starting with f1beab738665b378ebbcf2e410288bb8319fb96311abd002b601a7afbfe1004e not found: ID does not exist"
Nov 03 20:27:00.825140 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.825061    5282 scope.go:122] "RemoveContainer" containerID="4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467"
Nov 03 20:27:00.837812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.837589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:00.860635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.860487    5282 scope.go:122] "RemoveContainer" containerID="4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467"
Nov 03 20:27:00.860923 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:00.860896    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467\": container with ID starting with 4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467 not found: ID does not exist" containerID="4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467"
Nov 03 20:27:00.861009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.860936    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467"} err="failed to get container status \"4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467\": rpc error: code = NotFound desc = could not find container \"4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467\": container with ID starting with 4d109ac1fdb8a6d398b242cc2217f6a2665ac144ea0cee23ec7a5d32207bd467 not found: ID does not exist"
Nov 03 20:27:00.862276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862253    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-token\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-token\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862320    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-envoy\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862354    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-podinfo\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862418    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-data\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862509    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istiod-ca-cert\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862567    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-certs\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862628    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-credential-socket\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862678    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-kube-api-access-94gsj\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-kube-api-access-94gsj\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.862743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.862723    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-socket\") pod \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\" (UID: \"e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3\") "
Nov 03 20:27:00.863318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.863277    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-certs" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:00.863457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.863432    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-data" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:00.863566 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.863543    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-credential-socket" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:00.863648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.863585    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istiod-ca-cert" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:00.863648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.863603    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-socket" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:00.865018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.864993    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-podinfo" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:00.865296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.865268    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-envoy" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:00.865815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.865796    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-token" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:00.865886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.865866    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-kube-api-access-94gsj" pod "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" (UID: "e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3"). InnerVolumeSpecName "kube-api-access-94gsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:00.872480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.872455    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"]
Nov 03 20:27:00.881270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.881243    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-120fe3-65c4ff79b-4kwqh"]
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963852    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963880    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963890    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963901    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94gsj\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-kube-api-access-94gsj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963910    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963919    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963927    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.963931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963936    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:00.964381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:00.963945    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:01.133154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:01.133114    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"]
Nov 03 20:27:01.140743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:01.140710    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-456f55-openshift-default-5b9574668f-dkx8x"]
Nov 03 20:27:01.454052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:01.454019    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777432e5-8d85-41bc-b1d3-851ee8926850" path="/var/lib/kubelet/pods/777432e5-8d85-41bc-b1d3-851ee8926850/volumes"
Nov 03 20:27:01.454519 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:01.454504    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" path="/var/lib/kubelet/pods/e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3/volumes"
Nov 03 20:27:01.528007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:01.527976    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:01.615658 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:01.615618    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:27:02.309280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:02.309244    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:02.954227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:02.954155    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:03.020262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.020219    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main" containerID="cri-o://042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326" gracePeriod=45
Nov 03 20:27:03.044626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.044592    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main" containerID="cri-o://482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620" gracePeriod=45
Nov 03 20:27:03.216430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.215334    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:03.355826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.355793    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:27:03.374047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.374018    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:27:03.493997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.493969    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40ae433-7695-44d1-ba58-0614ce2aa2f5-kube-api-access-np7kv\" (UniqueName: \"kubernetes.io/projected/e40ae433-7695-44d1-ba58-0614ce2aa2f5-kube-api-access-np7kv\") pod \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") "
Nov 03 20:27:03.494226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494018    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-dshm\") pod \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") "
Nov 03 20:27:03.494226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494057    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-dshm\") pod \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") "
Nov 03 20:27:03.494226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494090    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tls-certs\") pod \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") "
Nov 03 20:27:03.494226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494126    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tmp-dir\") pod \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") "
Nov 03 20:27:03.494450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494309    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-kube-api-access-2lfkz\" (UniqueName: \"kubernetes.io/projected/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-kube-api-access-2lfkz\") pod \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") "
Nov 03 20:27:03.494450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494356    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tls-certs\") pod \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") "
Nov 03 20:27:03.494450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494425    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tmp-dir" pod "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" (UID: "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.494450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494435    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-home\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-home\") pod \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") "
Nov 03 20:27:03.494643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494502    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-home\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-home\") pod \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") "
Nov 03 20:27:03.494643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494584    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tmp-dir\") pod \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") "
Nov 03 20:27:03.494735 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494643    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-model-cache\") pod \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\" (UID: \"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947\") "
Nov 03 20:27:03.494891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494798    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-model-cache\") pod \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\" (UID: \"e40ae433-7695-44d1-ba58-0614ce2aa2f5\") "
Nov 03 20:27:03.494891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.494831    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-home" pod "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" (UID: "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.495064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495043    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tmp-dir" pod "e40ae433-7695-44d1-ba58-0614ce2aa2f5" (UID: "e40ae433-7695-44d1-ba58-0614ce2aa2f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.495305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495281    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-model-cache" pod "e40ae433-7695-44d1-ba58-0614ce2aa2f5" (UID: "e40ae433-7695-44d1-ba58-0614ce2aa2f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.495415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495396    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.495484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495447    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.495484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495465    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.495484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495479    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.495658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495480    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-home" pod "e40ae433-7695-44d1-ba58-0614ce2aa2f5" (UID: "e40ae433-7695-44d1-ba58-0614ce2aa2f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.495721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.495667    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-model-cache" pod "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" (UID: "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.496932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.496895    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tls-certs" pod "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" (UID: "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:03.497115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.497092    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-dshm" pod "e40ae433-7695-44d1-ba58-0614ce2aa2f5" (UID: "e40ae433-7695-44d1-ba58-0614ce2aa2f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.497115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.497101    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40ae433-7695-44d1-ba58-0614ce2aa2f5-kube-api-access-np7kv" pod "e40ae433-7695-44d1-ba58-0614ce2aa2f5" (UID: "e40ae433-7695-44d1-ba58-0614ce2aa2f5"). InnerVolumeSpecName "kube-api-access-np7kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:03.497115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.497109    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tls-certs" pod "e40ae433-7695-44d1-ba58-0614ce2aa2f5" (UID: "e40ae433-7695-44d1-ba58-0614ce2aa2f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:03.497317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.497098    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-dshm" pod "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" (UID: "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:03.497666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.497645    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-kube-api-access-2lfkz" pod "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" (UID: "4d36fc54-8a2c-45c0-9b1a-b7d441c1d947"). InnerVolumeSpecName "kube-api-access-2lfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:03.596553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596519    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np7kv\" (UniqueName: \"kubernetes.io/projected/e40ae433-7695-44d1-ba58-0614ce2aa2f5-kube-api-access-np7kv\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.596553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596550    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.596553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596562    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.596809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596575    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e40ae433-7695-44d1-ba58-0614ce2aa2f5-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.596809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596588    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lfkz\" (UniqueName: \"kubernetes.io/projected/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-kube-api-access-2lfkz\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.596809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596599    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.596809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596611    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e40ae433-7695-44d1-ba58-0614ce2aa2f5-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.596809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.596622    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:03.622318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.622280    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" podUID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:27:03.816525 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:03.816435    5282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4e8061_b105_45fc_8a9e_a35605b4bcb7.slice/crio-conmon-796caddb2755bde3c46684bdcc09a2110e5185ecb1bc47596a22f10321c3f9d1.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:27:03.836979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.836943    5282 generic.go:356] "Generic (PLEG): container finished" podID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerID="482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620" exitCode=0
Nov 03 20:27:03.837173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.837011    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"
Nov 03 20:27:03.837173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.837027    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" event={"ID":"e40ae433-7695-44d1-ba58-0614ce2aa2f5","Type":"ContainerDied","Data":"482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620"}
Nov 03 20:27:03.837173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.837075    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g" event={"ID":"e40ae433-7695-44d1-ba58-0614ce2aa2f5","Type":"ContainerDied","Data":"80ed36d14c31e651d221cd0d2d81fc44cdae297fb1f03fcd7c18119f3d7f5b70"}
Nov 03 20:27:03.837173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.837098    5282 scope.go:122] "RemoveContainer" containerID="482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620"
Nov 03 20:27:03.838506 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.838473    5282 generic.go:356] "Generic (PLEG): container finished" podID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerID="042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326" exitCode=0
Nov 03 20:27:03.838623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.838520    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" event={"ID":"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947","Type":"ContainerDied","Data":"042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326"}
Nov 03 20:27:03.838623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.838541    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"
Nov 03 20:27:03.838714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.838542    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l" event={"ID":"4d36fc54-8a2c-45c0-9b1a-b7d441c1d947","Type":"ContainerDied","Data":"6966f5fb6e814c12fcaff10b1a9d9e433f2cc93f6e22e02e1807baa5efe1bce5"}
Nov 03 20:27:03.846610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.846588    5282 scope.go:122] "RemoveContainer" containerID="482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620"
Nov 03 20:27:03.846982 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:03.846959    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620\": container with ID starting with 482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620 not found: ID does not exist" containerID="482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620"
Nov 03 20:27:03.847087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.846988    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620"} err="failed to get container status \"482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620\": rpc error: code = NotFound desc = could not find container \"482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620\": container with ID starting with 482d740d84c589c8ba525cfc0973355e9fd03a40aab0de54b508f10333c44620 not found: ID does not exist"
Nov 03 20:27:03.847087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.847005    5282 scope.go:122] "RemoveContainer" containerID="042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326"
Nov 03 20:27:03.855805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.855784    5282 scope.go:122] "RemoveContainer" containerID="042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326"
Nov 03 20:27:03.856078 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:03.856059    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326\": container with ID starting with 042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326 not found: ID does not exist" containerID="042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326"
Nov 03 20:27:03.856150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.856082    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326"} err="failed to get container status \"042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326\": rpc error: code = NotFound desc = could not find container \"042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326\": container with ID starting with 042da93d9b4daa9389dbfaa2e3708b34bf03aaad4e380b359d9463eb1ed77326 not found: ID does not exist"
Nov 03 20:27:03.865580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.865556    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"]
Nov 03 20:27:03.869590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.869564    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/premium-w5-ab1e46-kserve-7b44bfff68-hjx2g"]
Nov 03 20:27:03.881474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.881453    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"]
Nov 03 20:27:03.885439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.885417    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ab1e46/distinct-w5-ab1e46-kserve-5dfbfd56dc-nnd7l"]
Nov 03 20:27:03.894090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:03.894047    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:04.591925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:04.591884    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:04.798951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:04.798908    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" containerID="cri-o://048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed" gracePeriod=45
Nov 03 20:27:04.804567 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:04.804537    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main" containerID="cri-o://e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853" gracePeriod=45
Nov 03 20:27:04.809386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:04.809358    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Nov 03 20:27:05.058138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.058110    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:27:05.101084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.101014    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:27:05.130886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.130804    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-model-cache\") pod \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") "
Nov 03 20:27:05.130886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.130856    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tls-certs\") pod \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") "
Nov 03 20:27:05.131119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.130888    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-dshm\") pod \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") "
Nov 03 20:27:05.131119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.130918    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-model-cache\") pod \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") "
Nov 03 20:27:05.131119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131020    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tmp-dir\") pod \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") "
Nov 03 20:27:05.131119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131045    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-home\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-home\") pod \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") "
Nov 03 20:27:05.131119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131076    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-kube-api-access-r7m8h\" (UniqueName: \"kubernetes.io/projected/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-kube-api-access-r7m8h\") pod \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") "
Nov 03 20:27:05.131486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131138    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tls-certs\") pod \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") "
Nov 03 20:27:05.131486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131174    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-dshm\") pod \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") "
Nov 03 20:27:05.131486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131264    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tmp-dir\") pod \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") "
Nov 03 20:27:05.131486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131272    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-model-cache" pod "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" (UID: "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.131486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131294    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ec540a-467b-4a58-962a-4b804ebfb2ca-kube-api-access-2r4ln\" (UniqueName: \"kubernetes.io/projected/d3ec540a-467b-4a58-962a-4b804ebfb2ca-kube-api-access-2r4ln\") pod \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\" (UID: \"d3ec540a-467b-4a58-962a-4b804ebfb2ca\") "
Nov 03 20:27:05.131486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131331    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-home\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-home\") pod \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\" (UID: \"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0\") "
Nov 03 20:27:05.131813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131561    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-model-cache" pod "d3ec540a-467b-4a58-962a-4b804ebfb2ca" (UID: "d3ec540a-467b-4a58-962a-4b804ebfb2ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.131813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131742    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-home" pod "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" (UID: "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.131926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131840    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.131926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131859    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.131926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.131872    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.133021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.132332    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tmp-dir" pod "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" (UID: "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.133021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.132773    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-home" pod "d3ec540a-467b-4a58-962a-4b804ebfb2ca" (UID: "d3ec540a-467b-4a58-962a-4b804ebfb2ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.133021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.132991    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tmp-dir" pod "d3ec540a-467b-4a58-962a-4b804ebfb2ca" (UID: "d3ec540a-467b-4a58-962a-4b804ebfb2ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.134416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.134386    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-dshm" pod "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" (UID: "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.138150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.136109    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tls-certs" pod "d3ec540a-467b-4a58-962a-4b804ebfb2ca" (UID: "d3ec540a-467b-4a58-962a-4b804ebfb2ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:05.138150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.136601    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ec540a-467b-4a58-962a-4b804ebfb2ca-kube-api-access-2r4ln" pod "d3ec540a-467b-4a58-962a-4b804ebfb2ca" (UID: "d3ec540a-467b-4a58-962a-4b804ebfb2ca"). InnerVolumeSpecName "kube-api-access-2r4ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:05.138150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.137881    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-dshm" pod "d3ec540a-467b-4a58-962a-4b804ebfb2ca" (UID: "d3ec540a-467b-4a58-962a-4b804ebfb2ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:05.138150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.137929    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-kube-api-access-r7m8h" pod "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" (UID: "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0"). InnerVolumeSpecName "kube-api-access-r7m8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:05.138150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.138033    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tls-certs" pod "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" (UID: "3b82d548-716d-4a3e-98f1-0b3c8e4c91b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:05.232511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232471    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232500    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232510    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7m8h\" (UniqueName: \"kubernetes.io/projected/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-kube-api-access-r7m8h\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232521    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232531    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232539    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232547    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r4ln\" (UniqueName: \"kubernetes.io/projected/d3ec540a-467b-4a58-962a-4b804ebfb2ca-kube-api-access-2r4ln\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232557    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ec540a-467b-4a58-962a-4b804ebfb2ca-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.232732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.232564    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ec540a-467b-4a58-962a-4b804ebfb2ca-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:05.314609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.314559    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:05.454745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.454663    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" path="/var/lib/kubelet/pods/4d36fc54-8a2c-45c0-9b1a-b7d441c1d947/volumes"
Nov 03 20:27:05.455088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.455074    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" path="/var/lib/kubelet/pods/e40ae433-7695-44d1-ba58-0614ce2aa2f5/volumes"
Nov 03 20:27:05.853145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.853111    5282 generic.go:356] "Generic (PLEG): container finished" podID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerID="048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed" exitCode=0
Nov 03 20:27:05.853627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.853153    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"
Nov 03 20:27:05.853627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.853217    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" event={"ID":"d3ec540a-467b-4a58-962a-4b804ebfb2ca","Type":"ContainerDied","Data":"048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed"}
Nov 03 20:27:05.853627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.853264    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps" event={"ID":"d3ec540a-467b-4a58-962a-4b804ebfb2ca","Type":"ContainerDied","Data":"22b7113ba34c0a42ce21e438436bc477a7458b880f09eef6a30005dc11db1e25"}
Nov 03 20:27:05.853627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.853289    5282 scope.go:122] "RemoveContainer" containerID="048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed"
Nov 03 20:27:05.854679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.854653    5282 generic.go:356] "Generic (PLEG): container finished" podID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerID="e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853" exitCode=0
Nov 03 20:27:05.854791 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.854695    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"
Nov 03 20:27:05.854843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.854694    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" event={"ID":"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0","Type":"ContainerDied","Data":"e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853"}
Nov 03 20:27:05.854843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.854824    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52" event={"ID":"3b82d548-716d-4a3e-98f1-0b3c8e4c91b0","Type":"ContainerDied","Data":"56b2db2588d163d7a673f6da820d5cf94521924a98e6a67668ba2b93b1596a4d"}
Nov 03 20:27:05.862720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.862693    5282 scope.go:122] "RemoveContainer" containerID="048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed"
Nov 03 20:27:05.862994 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:05.862973    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed\": container with ID starting with 048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed not found: ID does not exist" containerID="048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed"
Nov 03 20:27:05.863040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.863001    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed"} err="failed to get container status \"048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed\": rpc error: code = NotFound desc = could not find container \"048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed\": container with ID starting with 048c707e4b99130fd729ca26afa04591152d9e1a4a72712bca388736197ea4ed not found: ID does not exist"
Nov 03 20:27:05.863040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.863017    5282 scope.go:122] "RemoveContainer" containerID="e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853"
Nov 03 20:27:05.870982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.870964    5282 scope.go:122] "RemoveContainer" containerID="e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853"
Nov 03 20:27:05.871261 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:05.871242    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853\": container with ID starting with e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853 not found: ID does not exist" containerID="e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853"
Nov 03 20:27:05.871309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.871270    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853"} err="failed to get container status \"e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853\": rpc error: code = NotFound desc = could not find container \"e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853\": container with ID starting with e0582ac39a90bd404e8c97838a054ee0c5c49e6cef76d31f3fb474cc56d1f853 not found: ID does not exist"
Nov 03 20:27:05.875161 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.875138    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"]
Nov 03 20:27:05.879115 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.879095    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-w1-b42aea-kserve-b88b7f8fc-msb52"]
Nov 03 20:27:05.888482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.888461    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"]
Nov 03 20:27:05.894572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.894553    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b42aea/distinct-2-w1-b42aea-kserve-867c666cd6-67kps"]
Nov 03 20:27:05.985898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:05.985866    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.069945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:06.069909    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"]
Nov 03 20:27:06.168127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:06.168045    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.293919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:06.293881    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.565039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:06.565010    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.706236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:06.702675    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.827876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:06.827782    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:06.946871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:06.946836    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:07.267923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:07.267890    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:07.455414 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:07.455381    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" path="/var/lib/kubelet/pods/3b82d548-716d-4a3e-98f1-0b3c8e4c91b0/volumes"
Nov 03 20:27:07.455862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:07.455842    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" path="/var/lib/kubelet/pods/d3ec540a-467b-4a58-962a-4b804ebfb2ca/volumes"
Nov 03 20:27:07.613820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:07.613734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:07.985984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:07.985947    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:08.163399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:08.163359    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:08.171794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:08.171757    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"]
Nov 03 20:27:08.172030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:08.172005    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy" containerID="cri-o://f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65" gracePeriod=30
Nov 03 20:27:08.320298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:08.319088    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:08.518817 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:08.518759    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:09.246686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.245641    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:09.698950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.698848    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"]
Nov 03 20:27:09.699154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.699073    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv" podUID="0d433013-5364-4ebe-96d9-08e53103e387" containerName="maas-api" containerID="cri-o://1329b206448ec3d8418ccbaf21f3cc0cec363eed314564c421a8fec255d79650" gracePeriod=30
Nov 03 20:27:09.792876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.792846    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:09.886808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.886771    5282 generic.go:356] "Generic (PLEG): container finished" podID="0d433013-5364-4ebe-96d9-08e53103e387" containerID="1329b206448ec3d8418ccbaf21f3cc0cec363eed314564c421a8fec255d79650" exitCode=0
Nov 03 20:27:09.886994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.886834    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv" event={"ID":"0d433013-5364-4ebe-96d9-08e53103e387","Type":"ContainerDied","Data":"1329b206448ec3d8418ccbaf21f3cc0cec363eed314564c421a8fec255d79650"}
Nov 03 20:27:09.913223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.913168    5282 scope.go:122] "RemoveContainer" containerID="72a6485d80472dea967a4cc228bffdd476cf5f446a5f34841be9fe5049bf8f60"
Nov 03 20:27:09.925019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.924985    5282 scope.go:122] "RemoveContainer" containerID="c1944a589cede9a8524a14036b574e5ee655400e60cb4454341c78e27e9523ba"
Nov 03 20:27:09.929835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.929810    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:09.970252 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:09.970225    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:27:10.050815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.050789    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:10.099524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.098643    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d433013-5364-4ebe-96d9-08e53103e387-kube-api-access-7qblr\" (UniqueName: \"kubernetes.io/projected/0d433013-5364-4ebe-96d9-08e53103e387-kube-api-access-7qblr\") pod \"0d433013-5364-4ebe-96d9-08e53103e387\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") "
Nov 03 20:27:10.099524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.098723    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-maas-api-tls\") pod \"0d433013-5364-4ebe-96d9-08e53103e387\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") "
Nov 03 20:27:10.099524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.098813    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d433013-5364-4ebe-96d9-08e53103e387-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d433013-5364-4ebe-96d9-08e53103e387-openshift-service-ca\") pod \"0d433013-5364-4ebe-96d9-08e53103e387\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") "
Nov 03 20:27:10.099524 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.098872    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-metrics-certs\") pod \"0d433013-5364-4ebe-96d9-08e53103e387\" (UID: \"0d433013-5364-4ebe-96d9-08e53103e387\") "
Nov 03 20:27:10.112035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.102864    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-maas-api-tls" pod "0d433013-5364-4ebe-96d9-08e53103e387" (UID: "0d433013-5364-4ebe-96d9-08e53103e387"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:10.118545 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.117361    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d433013-5364-4ebe-96d9-08e53103e387-kube-api-access-7qblr" pod "0d433013-5364-4ebe-96d9-08e53103e387" (UID: "0d433013-5364-4ebe-96d9-08e53103e387"). InnerVolumeSpecName "kube-api-access-7qblr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:10.119498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.119428    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d433013-5364-4ebe-96d9-08e53103e387-openshift-service-ca" pod "0d433013-5364-4ebe-96d9-08e53103e387" (UID: "0d433013-5364-4ebe-96d9-08e53103e387"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:10.141628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.139365    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-metrics-certs" pod "0d433013-5364-4ebe-96d9-08e53103e387" (UID: "0d433013-5364-4ebe-96d9-08e53103e387"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:10.178538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.178499    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:10.200573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.200495    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:10.200573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.200521    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qblr\" (UniqueName: \"kubernetes.io/projected/0d433013-5364-4ebe-96d9-08e53103e387-kube-api-access-7qblr\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:10.200573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.200533    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d433013-5364-4ebe-96d9-08e53103e387-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:10.200573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.200542    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d433013-5364-4ebe-96d9-08e53103e387-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:10.407814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.407779    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:10.892946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.892917    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"
Nov 03 20:27:10.893131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.892966    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv" event={"ID":"0d433013-5364-4ebe-96d9-08e53103e387","Type":"ContainerDied","Data":"efbc06f65427059eac6dd1a52a70f981b7e8800c0f2333e0949435e6b923dd4b"}
Nov 03 20:27:10.893131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.893062    5282 scope.go:122] "RemoveContainer" containerID="1329b206448ec3d8418ccbaf21f3cc0cec363eed314564c421a8fec255d79650"
Nov 03 20:27:10.893131 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.893080    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:10.939344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.939299    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"]
Nov 03 20:27:10.939579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:10.939368    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b42aea-84f4bf4dc7-zxlrv"]
Nov 03 20:27:11.070451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.070403    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss" podUID="ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" containerName="payload-processing" containerID="cri-o://6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f" gracePeriod=25
Nov 03 20:27:11.317143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.317119    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:27:11.414209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.414173    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-kube-api-access-rdxqv\" (UniqueName: \"kubernetes.io/projected/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-kube-api-access-rdxqv\") pod \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") "
Nov 03 20:27:11.414663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.414293    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-plugins-config-volume\") pod \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\" (UID: \"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba\") "
Nov 03 20:27:11.414727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.414684    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-plugins-config-volume" pod "ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" (UID: "ea7c63cf-607d-467c-9239-9ed4e1e3a6ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:11.416286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.416264    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-kube-api-access-rdxqv" pod "ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" (UID: "ea7c63cf-607d-467c-9239-9ed4e1e3a6ba"). InnerVolumeSpecName "kube-api-access-rdxqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:11.453902 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.453832    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d433013-5364-4ebe-96d9-08e53103e387" path="/var/lib/kubelet/pods/0d433013-5364-4ebe-96d9-08e53103e387/volumes"
Nov 03 20:27:11.515765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.515731    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxqv\" (UniqueName: \"kubernetes.io/projected/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-kube-api-access-rdxqv\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:11.515765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.515765    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:11.631016 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.628743    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:11.900017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.899984    5282 generic.go:356] "Generic (PLEG): container finished" podID="ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" containerID="6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f" exitCode=0
Nov 03 20:27:11.900242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.900024    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"
Nov 03 20:27:11.900242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.900058    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss" event={"ID":"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba","Type":"ContainerDied","Data":"6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f"}
Nov 03 20:27:11.900242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.900111    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss" event={"ID":"ea7c63cf-607d-467c-9239-9ed4e1e3a6ba","Type":"ContainerDied","Data":"3ac3b046e88b1fb9092b497aefecfd9226915bf8d44706cdd620b73923e816f5"}
Nov 03 20:27:11.900242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.900138    5282 scope.go:122] "RemoveContainer" containerID="6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f"
Nov 03 20:27:11.909327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.909305    5282 scope.go:122] "RemoveContainer" containerID="6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f"
Nov 03 20:27:11.909598 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:11.909581    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f\": container with ID starting with 6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f not found: ID does not exist" containerID="6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f"
Nov 03 20:27:11.909651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.909605    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f"} err="failed to get container status \"6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f\": rpc error: code = NotFound desc = could not find container \"6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f\": container with ID starting with 6e9fc12a8859404925434e1289e500352b1c29a87a179d5346cc580cc4920a6f not found: ID does not exist"
Nov 03 20:27:11.917929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.917903    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"]
Nov 03 20:27:11.921329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:11.921305    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b55105-5888d84df4-bk4ss"]
Nov 03 20:27:12.389333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:12.389302    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:13.139488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.139453    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:13.271875 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.271840    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:13.394321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.394245    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:13.454325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.454297    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" path="/var/lib/kubelet/pods/ea7c63cf-607d-467c-9239-9ed4e1e3a6ba/volumes"
Nov 03 20:27:13.460985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.460964    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:27:13.513162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.513129    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:13.542076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.542037    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-certs\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.543270 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.542797    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istiod-ca-cert\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.543951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.543932    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-envoy\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.544447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.542664    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-certs" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.544558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.544467    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-podinfo\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.544558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.544516    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-credential-socket\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.544689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.544615    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-kube-api-access-wzlkj\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-kube-api-access-wzlkj\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.544689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.544682    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-socket\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.544789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.544723    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-token\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.544789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.544763    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-data\") pod \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\" (UID: \"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa\") "
Nov 03 20:27:13.545785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.543861    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istiod-ca-cert" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:13.545785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.545477    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-socket" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.545785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.545510    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.547732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.546612    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-envoy" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.547732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.547490    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-data" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.547732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.547687    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-credential-socket" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:13.549638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.549604    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-podinfo" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:13.550833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.550799    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-kube-api-access-wzlkj" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "kube-api-access-wzlkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:13.553017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.551304    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-token" pod "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" (UID: "ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:13.622342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.622306    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" podUID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646669    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646698    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646708    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzlkj\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-kube-api-access-wzlkj\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646719    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646728    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646736    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646745    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.646750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.646752    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:13.913654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.913561    5282 generic.go:356] "Generic (PLEG): container finished" podID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerID="f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65" exitCode=0
Nov 03 20:27:13.913654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.913609    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" event={"ID":"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa","Type":"ContainerDied","Data":"f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65"}
Nov 03 20:27:13.913654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.913615    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"
Nov 03 20:27:13.913894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.913660    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z" event={"ID":"ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa","Type":"ContainerDied","Data":"879741170f4348c1108f234e522d86504f675bb80bdb3c41621b510b143eaf13"}
Nov 03 20:27:13.913894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.913693    5282 scope.go:122] "RemoveContainer" containerID="f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65"
Nov 03 20:27:13.922945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.922923    5282 scope.go:122] "RemoveContainer" containerID="f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65"
Nov 03 20:27:13.923248 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:13.923230    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65\": container with ID starting with f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65 not found: ID does not exist" containerID="f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65"
Nov 03 20:27:13.923327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.923255    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65"} err="failed to get container status \"f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65\": rpc error: code = NotFound desc = could not find container \"f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65\": container with ID starting with f8e32d19de1db6426e40ff4899f553340a439f283a88a2b46070c9a164f45b65 not found: ID does not exist"
Nov 03 20:27:13.935234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.935191    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"]
Nov 03 20:27:13.939901 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:13.939872    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-120fe3-openshift-default-cdb9b5cdc-z8f5z"]
Nov 03 20:27:14.156150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:14.156109    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:14.288472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:14.288439    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:14.902963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:14.902928    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:15.047830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:15.047770    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:15.176380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:15.176294    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"]
Nov 03 20:27:15.176836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:15.176513    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy" containerID="cri-o://a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664" gracePeriod=30
Nov 03 20:27:15.245208 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:15.245142    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:15.454351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:15.454272    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" path="/var/lib/kubelet/pods/ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa/volumes"
Nov 03 20:27:15.692813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:15.692773    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:16.064034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:16.063991    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:17.107463 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.107429    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:17.244694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.244652    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:17.391007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.390900    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"]
Nov 03 20:27:17.391191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.391163    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" containerID="cri-o://947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2" gracePeriod=30
Nov 03 20:27:17.490742 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.490703    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:17.798876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.798838    5282 patch_prober.go:29] interesting pod/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:17.799086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.798901    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:17.973840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:17.973799    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:18.608771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.608739    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:18.707515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.707454    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:27:18.786125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.786092    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:18.810484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.810450    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e978959-92f5-4c2a-afc3-91ae95b77dd9-kube-api-access-v4vrx\" (UniqueName: \"kubernetes.io/projected/4e978959-92f5-4c2a-afc3-91ae95b77dd9-kube-api-access-v4vrx\") pod \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") "
Nov 03 20:27:18.810666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.810569    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e978959-92f5-4c2a-afc3-91ae95b77dd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e978959-92f5-4c2a-afc3-91ae95b77dd9-plugins-config-volume\") pod \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\" (UID: \"4e978959-92f5-4c2a-afc3-91ae95b77dd9\") "
Nov 03 20:27:18.810983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.810957    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e978959-92f5-4c2a-afc3-91ae95b77dd9-plugins-config-volume" pod "4e978959-92f5-4c2a-afc3-91ae95b77dd9" (UID: "4e978959-92f5-4c2a-afc3-91ae95b77dd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:18.812605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.812579    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e978959-92f5-4c2a-afc3-91ae95b77dd9-kube-api-access-v4vrx" pod "4e978959-92f5-4c2a-afc3-91ae95b77dd9" (UID: "4e978959-92f5-4c2a-afc3-91ae95b77dd9"). InnerVolumeSpecName "kube-api-access-v4vrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:18.911868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.911826    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vrx\" (UniqueName: \"kubernetes.io/projected/4e978959-92f5-4c2a-afc3-91ae95b77dd9-kube-api-access-v4vrx\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:18.911868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.911863    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e978959-92f5-4c2a-afc3-91ae95b77dd9-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:18.961565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.961474    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"
Nov 03 20:27:18.961850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.961482    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" event={"ID":"4e978959-92f5-4c2a-afc3-91ae95b77dd9","Type":"ContainerDied","Data":"fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501"}
Nov 03 20:27:18.961962 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.961926    5282 scope.go:122] "RemoveContainer" containerID="fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501"
Nov 03 20:27:18.963145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.963116    5282 generic.go:356] "Generic (PLEG): container finished" podID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" containerID="fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501" exitCode=0
Nov 03 20:27:18.963954 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.963889    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69" event={"ID":"4e978959-92f5-4c2a-afc3-91ae95b77dd9","Type":"ContainerDied","Data":"8ee4920e87210cc60c31d849c899b6b1c6a66cef4a83aea2bf8fde17e2b24d6d"}
Nov 03 20:27:18.975511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.975459    5282 scope.go:122] "RemoveContainer" containerID="fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501"
Nov 03 20:27:18.975961 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:18.975934    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501\": container with ID starting with fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501 not found: ID does not exist" containerID="fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501"
Nov 03 20:27:18.976084 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.975972    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501"} err="failed to get container status \"fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501\": rpc error: code = NotFound desc = could not find container \"fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501\": container with ID starting with fab2c352b7c9a5dc08ad9418dba188bb6da5addbc8a48a6b1dd9181bb8117501 not found: ID does not exist"
Nov 03 20:27:18.977493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.977469    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"]
Nov 03 20:27:18.977708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:18.977684    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy" containerID="cri-o://beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a" gracePeriod=30
Nov 03 20:27:19.009724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:19.009681    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"]
Nov 03 20:27:19.026452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:19.026415    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ab1e46-5cb56fc799-8lk69"]
Nov 03 20:27:19.136049 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:19.136018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:19.454750 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:19.454711    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" path="/var/lib/kubelet/pods/4e978959-92f5-4c2a-afc3-91ae95b77dd9/volumes"
Nov 03 20:27:19.691578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:19.691541    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:20.393069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.393037    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:20.451258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.451168    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:27:20.536445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536409    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-kube-api-access-v5p52\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-kube-api-access-v5p52\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536474    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-credential-socket\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536509    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-certs\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536547    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-data\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536630    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-podinfo\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536670    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-token\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-token\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536708    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-envoy\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536798    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istiod-ca-cert\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536835    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-socket\") pod \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\" (UID: \"b253246c-0adb-4d4d-b6db-4bfa0c3993f4\") "
Nov 03 20:27:20.536968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536836    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-credential-socket" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.536968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536876    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-certs" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.536968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.536921    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-data" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.537223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.537181    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-socket" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.537223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.537213    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istiod-ca-cert" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:20.537355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.537226    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.537355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.537248    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.537355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.537262    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.539056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.539022    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-podinfo" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:20.539216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.539173    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-kube-api-access-v5p52" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "kube-api-access-v5p52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.539429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.539411    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-token" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:20.539510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.539491    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-envoy" pod "b253246c-0adb-4d4d-b6db-4bfa0c3993f4" (UID: "b253246c-0adb-4d4d-b6db-4bfa0c3993f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:20.615186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.615150    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"]
Nov 03 20:27:20.638136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.638095    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.638136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.638131    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.638376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.638145    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.638376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.638158    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5p52\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-kube-api-access-v5p52\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.638376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.638174    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.638376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.638185    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253246c-0adb-4d4d-b6db-4bfa0c3993f4-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:20.778063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.778015    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:20.899344 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.899309    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:20.977986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.977946    5282 generic.go:356] "Generic (PLEG): container finished" podID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerID="a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664" exitCode=0
Nov 03 20:27:20.978162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.977998    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"
Nov 03 20:27:20.978162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.978005    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" event={"ID":"b253246c-0adb-4d4d-b6db-4bfa0c3993f4","Type":"ContainerDied","Data":"a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664"}
Nov 03 20:27:20.978162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.978058    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn" event={"ID":"b253246c-0adb-4d4d-b6db-4bfa0c3993f4","Type":"ContainerDied","Data":"0bb84febe257b3b097a303f4f6f0e02680396eba2cc5938bc43840e911323e17"}
Nov 03 20:27:20.978162 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.978088    5282 scope.go:122] "RemoveContainer" containerID="a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664"
Nov 03 20:27:20.987887 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.987863    5282 scope.go:122] "RemoveContainer" containerID="a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664"
Nov 03 20:27:20.988148 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:20.988128    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664\": container with ID starting with a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664 not found: ID does not exist" containerID="a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664"
Nov 03 20:27:20.988267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:20.988156    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664"} err="failed to get container status \"a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664\": rpc error: code = NotFound desc = could not find container \"a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664\": container with ID starting with a49a62da748e29ca2d0a17e654b4ab7ebbfb363944710f16a53870679cf29664 not found: ID does not exist"
Nov 03 20:27:21.002268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:21.002231    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"]
Nov 03 20:27:21.005503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:21.005481    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b55105-openshift-default-df8cb58b9-zg8nn"]
Nov 03 20:27:21.236754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:21.236111    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:21.453984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:21.453952    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" path="/var/lib/kubelet/pods/b253246c-0adb-4d4d-b6db-4bfa0c3993f4/volumes"
Nov 03 20:27:22.010429 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.010396    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:22.133092 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.133054    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:22.647362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.647334    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:27:22.703299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.703210    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:22.757573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757537    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8e93e9-5642-420b-96df-068346841dd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8e93e9-5642-420b-96df-068346841dd4-istiod-ca-cert\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.757760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757607    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-kube-api-access-cs7n7\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.757760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757638    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-socket\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.757760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757672    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff8e93e9-5642-420b-96df-068346841dd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8e93e9-5642-420b-96df-068346841dd4-istio-podinfo\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.757760 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757701    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-istio-token\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-istio-token\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.757982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757759    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-envoy\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.757982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757808    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-certs\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.757982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.757962    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8e93e9-5642-420b-96df-068346841dd4-istiod-ca-cert" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:22.758157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758008    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-socket" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.758157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758096    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-certs" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.758157 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758129    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-data\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.758400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758252    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-credential-socket\") pod \"ff8e93e9-5642-420b-96df-068346841dd4\" (UID: \"ff8e93e9-5642-420b-96df-068346841dd4\") "
Nov 03 20:27:22.758400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758311    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-data" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.758517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758482    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-credential-socket" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.758915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758891    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.759046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758946    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.759046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758964    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.759046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758981    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff8e93e9-5642-420b-96df-068346841dd4-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.759046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.758998    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.760129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.760066    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-istio-token" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:22.760129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.760111    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-kube-api-access-cs7n7" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "kube-api-access-cs7n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:22.760306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.760144    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-envoy" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:22.760306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.760153    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff8e93e9-5642-420b-96df-068346841dd4-istio-podinfo" pod "ff8e93e9-5642-420b-96df-068346841dd4" (UID: "ff8e93e9-5642-420b-96df-068346841dd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:22.859961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.859919    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-kube-api-access-cs7n7\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.859961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.859954    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff8e93e9-5642-420b-96df-068346841dd4-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.859961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.859967    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff8e93e9-5642-420b-96df-068346841dd4-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.860225 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.859975    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff8e93e9-5642-420b-96df-068346841dd4-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:22.989777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.989746    5282 generic.go:356] "Generic (PLEG): container finished" podID="ff8e93e9-5642-420b-96df-068346841dd4" containerID="947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2" exitCode=0
Nov 03 20:27:22.989994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.989793    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"
Nov 03 20:27:22.989994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.989835    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" event={"ID":"ff8e93e9-5642-420b-96df-068346841dd4","Type":"ContainerDied","Data":"947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2"}
Nov 03 20:27:22.989994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.989928    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv" event={"ID":"ff8e93e9-5642-420b-96df-068346841dd4","Type":"ContainerDied","Data":"20b7b76ddd5607d1b8891a405958eb4fbdb30c8e3c70a32039e9f2fc1aaa6684"}
Nov 03 20:27:22.989994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.989957    5282 scope.go:122] "RemoveContainer" containerID="947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2"
Nov 03 20:27:22.999538 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.999521    5282 scope.go:122] "RemoveContainer" containerID="947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2"
Nov 03 20:27:22.999800 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:22.999782    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2\": container with ID starting with 947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2 not found: ID does not exist" containerID="947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2"
Nov 03 20:27:22.999850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:22.999808    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2"} err="failed to get container status \"947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2\": rpc error: code = NotFound desc = could not find container \"947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2\": container with ID starting with 947327eb17cb361ccd0002fd733bd40dc2f50d7e0e59582933f7ce37ca3905c2 not found: ID does not exist"
Nov 03 20:27:23.011675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:23.011649    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"]
Nov 03 20:27:23.016003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:23.015978    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ab1e46-openshift-default-568c5c7fbf-rhtcv"]
Nov 03 20:27:23.416000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:23.415922    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:23.454456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:23.454417    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8e93e9-5642-420b-96df-068346841dd4" path="/var/lib/kubelet/pods/ff8e93e9-5642-420b-96df-068346841dd4/volumes"
Nov 03 20:27:24.231912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.231879    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:27:24.279610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.279529    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:27:24.306022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.305987    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:24.375350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375279    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-token\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-token\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.375585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375363    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-kube-api-access-92nvt\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-kube-api-access-92nvt\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.375585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375413    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-credential-socket\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.375747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375704    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-credential-socket" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.375808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375776    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istiod-ca-cert\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.375862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375834    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-data\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.375928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375913    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-envoy\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.375979 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.375949    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-socket\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.376083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376020    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-certs\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.376152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376137    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-podinfo\") pod \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\" (UID: \"3416cbfd-53c7-494a-81e5-ec52c63d39d6\") "
Nov 03 20:27:24.376188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376163    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-data" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.376290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376266    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istiod-ca-cert" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:24.376417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376395    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-socket" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.376605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376581    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-certs" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.376690 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376672    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.376759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376695    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.376759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376709    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.376759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376724    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.376759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.376740    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.378007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.377985    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-token" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:24.378082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.378054    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-kube-api-access-92nvt" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "kube-api-access-92nvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:24.378351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.378331    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-envoy" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:24.378551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.378533    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-podinfo" pod "3416cbfd-53c7-494a-81e5-ec52c63d39d6" (UID: "3416cbfd-53c7-494a-81e5-ec52c63d39d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:24.477295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.477260    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.477295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.477294    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.477521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.477310    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.477521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.477322    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nvt\" (UniqueName: \"kubernetes.io/projected/3416cbfd-53c7-494a-81e5-ec52c63d39d6-kube-api-access-92nvt\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:24.692904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.692825    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:24.815527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:24.815485    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:25.003124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.003082    5282 generic.go:356] "Generic (PLEG): container finished" podID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerID="beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a" exitCode=0
Nov 03 20:27:25.003316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.003139    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"
Nov 03 20:27:25.003316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.003157    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" event={"ID":"3416cbfd-53c7-494a-81e5-ec52c63d39d6","Type":"ContainerDied","Data":"beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a"}
Nov 03 20:27:25.003316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.003226    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh" event={"ID":"3416cbfd-53c7-494a-81e5-ec52c63d39d6","Type":"ContainerDied","Data":"160acb3db6caa4f427da0f82a2f71b36bd81b8ade037ba5cd89efdca2f3d7929"}
Nov 03 20:27:25.003316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.003258    5282 scope.go:122] "RemoveContainer" containerID="beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a"
Nov 03 20:27:25.013032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.013014    5282 scope.go:122] "RemoveContainer" containerID="beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a"
Nov 03 20:27:25.013364 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:25.013341    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a\": container with ID starting with beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a not found: ID does not exist" containerID="beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a"
Nov 03 20:27:25.013447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.013373    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a"} err="failed to get container status \"beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a\": rpc error: code = NotFound desc = could not find container \"beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a\": container with ID starting with beb0492b68ddde53a54e4a646247358f4648c54aaae6191bf6585ec48ef51b0a not found: ID does not exist"
Nov 03 20:27:25.024667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.024641    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"]
Nov 03 20:27:25.027730 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.027699    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b42aea-openshift-default-7964669b97-nkqnh"]
Nov 03 20:27:25.089967 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.089638    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:25.453905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.453807    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" path="/var/lib/kubelet/pods/3416cbfd-53c7-494a-81e5-ec52c63d39d6/volumes"
Nov 03 20:27:25.880614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.880574    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:25.925015 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.924976    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"]
Nov 03 20:27:25.940230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:25.938978    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"]
Nov 03 20:27:26.118073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:26.118042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:26.369868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:26.369826    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:26.716707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:26.716625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:28.190903 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:28.190862    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:28.308957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:28.308923    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:28.420343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:28.420308    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:28.536288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:28.536259    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:28.765097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:28.765066    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:30.926054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:30.925984    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd" podUID="a7b2bd90-9bb5-469d-a8cb-194f786a1469" containerName="payload-processing" containerID="cri-o://d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d" gracePeriod=25
Nov 03 20:27:30.940280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:30.940244    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd" podUID="f866abde-bff2-407c-918e-fa4aa493084c" containerName="payload-pre-processing" containerID="cri-o://3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611" gracePeriod=25
Nov 03 20:27:31.193551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.193485    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:27:31.224430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.224402    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:27:31.349468 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.349430    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b2bd90-9bb5-469d-a8cb-194f786a1469-kube-api-access-l5f2w\" (UniqueName: \"kubernetes.io/projected/a7b2bd90-9bb5-469d-a8cb-194f786a1469-kube-api-access-l5f2w\") pod \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") "
Nov 03 20:27:31.349632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.349477    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b2bd90-9bb5-469d-a8cb-194f786a1469-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b2bd90-9bb5-469d-a8cb-194f786a1469-plugins-config-volume\") pod \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\" (UID: \"a7b2bd90-9bb5-469d-a8cb-194f786a1469\") "
Nov 03 20:27:31.349632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.349528    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f866abde-bff2-407c-918e-fa4aa493084c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f866abde-bff2-407c-918e-fa4aa493084c-plugins-config-volume\") pod \"f866abde-bff2-407c-918e-fa4aa493084c\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") "
Nov 03 20:27:31.349632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.349607    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f866abde-bff2-407c-918e-fa4aa493084c-kube-api-access-t944b\" (UniqueName: \"kubernetes.io/projected/f866abde-bff2-407c-918e-fa4aa493084c-kube-api-access-t944b\") pod \"f866abde-bff2-407c-918e-fa4aa493084c\" (UID: \"f866abde-bff2-407c-918e-fa4aa493084c\") "
Nov 03 20:27:31.349916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.349890    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b2bd90-9bb5-469d-a8cb-194f786a1469-plugins-config-volume" pod "a7b2bd90-9bb5-469d-a8cb-194f786a1469" (UID: "a7b2bd90-9bb5-469d-a8cb-194f786a1469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:31.349985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.349960    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f866abde-bff2-407c-918e-fa4aa493084c-plugins-config-volume" pod "f866abde-bff2-407c-918e-fa4aa493084c" (UID: "f866abde-bff2-407c-918e-fa4aa493084c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:31.351708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.351685    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f866abde-bff2-407c-918e-fa4aa493084c-kube-api-access-t944b" pod "f866abde-bff2-407c-918e-fa4aa493084c" (UID: "f866abde-bff2-407c-918e-fa4aa493084c"). InnerVolumeSpecName "kube-api-access-t944b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:31.351776 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.351762    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b2bd90-9bb5-469d-a8cb-194f786a1469-kube-api-access-l5f2w" pod "a7b2bd90-9bb5-469d-a8cb-194f786a1469" (UID: "a7b2bd90-9bb5-469d-a8cb-194f786a1469"). InnerVolumeSpecName "kube-api-access-l5f2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:31.450751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.450670    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f866abde-bff2-407c-918e-fa4aa493084c-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:31.450751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.450698    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t944b\" (UniqueName: \"kubernetes.io/projected/f866abde-bff2-407c-918e-fa4aa493084c-kube-api-access-t944b\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:31.450751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.450708    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5f2w\" (UniqueName: \"kubernetes.io/projected/a7b2bd90-9bb5-469d-a8cb-194f786a1469-kube-api-access-l5f2w\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:31.450751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:31.450717    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7b2bd90-9bb5-469d-a8cb-194f786a1469-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:31.614344 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:31.614301    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:27:32.040447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.040414    5282 generic.go:356] "Generic (PLEG): container finished" podID="a7b2bd90-9bb5-469d-a8cb-194f786a1469" containerID="d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d" exitCode=0
Nov 03 20:27:32.040923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.040456    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"
Nov 03 20:27:32.040923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.040489    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd" event={"ID":"a7b2bd90-9bb5-469d-a8cb-194f786a1469","Type":"ContainerDied","Data":"d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d"}
Nov 03 20:27:32.040923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.040547    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd" event={"ID":"a7b2bd90-9bb5-469d-a8cb-194f786a1469","Type":"ContainerDied","Data":"5bd8eb1b9beb7da99d439d5fb7b5ec4641829bc63ba1338e3d244f7e6c8b7313"}
Nov 03 20:27:32.040923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.040576    5282 scope.go:122] "RemoveContainer" containerID="d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d"
Nov 03 20:27:32.041953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.041825    5282 generic.go:356] "Generic (PLEG): container finished" podID="f866abde-bff2-407c-918e-fa4aa493084c" containerID="3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611" exitCode=0
Nov 03 20:27:32.041953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.041883    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"
Nov 03 20:27:32.041953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.041919    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd" event={"ID":"f866abde-bff2-407c-918e-fa4aa493084c","Type":"ContainerDied","Data":"3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611"}
Nov 03 20:27:32.042163 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.041972    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd" event={"ID":"f866abde-bff2-407c-918e-fa4aa493084c","Type":"ContainerDied","Data":"d6e6cf154739f48030c889ac2b1893443761182af5f75f1b6a9a57708dfd8e9f"}
Nov 03 20:27:32.049605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.049588    5282 scope.go:122] "RemoveContainer" containerID="d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d"
Nov 03 20:27:32.049906 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:32.049887    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d\": container with ID starting with d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d not found: ID does not exist" containerID="d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d"
Nov 03 20:27:32.049955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.049914    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d"} err="failed to get container status \"d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d\": rpc error: code = NotFound desc = could not find container \"d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d\": container with ID starting with d049f039e5ab25523227a41eba2e96ed825ba74095da22e9721f55f5f1a1151d not found: ID does not exist"
Nov 03 20:27:32.049955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.049932    5282 scope.go:122] "RemoveContainer" containerID="3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611"
Nov 03 20:27:32.057904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.057876    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"]
Nov 03 20:27:32.058789 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.058774    5282 scope.go:122] "RemoveContainer" containerID="3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611"
Nov 03 20:27:32.059145 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:32.059119    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611\": container with ID starting with 3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611 not found: ID does not exist" containerID="3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611"
Nov 03 20:27:32.059226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.059154    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611"} err="failed to get container status \"3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611\": rpc error: code = NotFound desc = could not find container \"3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611\": container with ID starting with 3b9be1a30622904720d6b1782b88af3b538734b61a91f8d7875a20c712997611 not found: ID does not exist"
Nov 03 20:27:32.064981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.064953    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-11a5b0-5b9765647f-2p5zd"]
Nov 03 20:27:32.080409 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.080380    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"]
Nov 03 20:27:32.084310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.084288    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-11a5b0-6f4777d64b-hsxnd"]
Nov 03 20:27:32.386147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.386101    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:32.567155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:32.567092    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:33.453551 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:33.453522    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b2bd90-9bb5-469d-a8cb-194f786a1469" path="/var/lib/kubelet/pods/a7b2bd90-9bb5-469d-a8cb-194f786a1469/volumes"
Nov 03 20:27:33.453936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:33.453895    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f866abde-bff2-407c-918e-fa4aa493084c" path="/var/lib/kubelet/pods/f866abde-bff2-407c-918e-fa4aa493084c/volumes"
Nov 03 20:27:34.374173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:34.374141    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:35.635564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.635522    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main" containerID="cri-o://b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d" gracePeriod=45
Nov 03 20:27:35.887187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.887122    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:27:35.992106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992068    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-tmp-dir\") pod \"885dd171-716d-4d0f-a798-2072f63c243c\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") "
Nov 03 20:27:35.992304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992114    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-home\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-home\") pod \"885dd171-716d-4d0f-a798-2072f63c243c\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") "
Nov 03 20:27:35.992304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992133    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/885dd171-716d-4d0f-a798-2072f63c243c-tls-certs\" (UniqueName: \"kubernetes.io/secret/885dd171-716d-4d0f-a798-2072f63c243c-tls-certs\") pod \"885dd171-716d-4d0f-a798-2072f63c243c\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") "
Nov 03 20:27:35.992304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992271    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885dd171-716d-4d0f-a798-2072f63c243c-kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/885dd171-716d-4d0f-a798-2072f63c243c-kube-api-access-hg92x\") pod \"885dd171-716d-4d0f-a798-2072f63c243c\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") "
Nov 03 20:27:35.992506 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992359    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-model-cache\") pod \"885dd171-716d-4d0f-a798-2072f63c243c\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") "
Nov 03 20:27:35.992506 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992390    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-dshm\") pod \"885dd171-716d-4d0f-a798-2072f63c243c\" (UID: \"885dd171-716d-4d0f-a798-2072f63c243c\") "
Nov 03 20:27:35.992506 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992447    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-home" pod "885dd171-716d-4d0f-a798-2072f63c243c" (UID: "885dd171-716d-4d0f-a798-2072f63c243c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:35.992700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992466    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-tmp-dir" pod "885dd171-716d-4d0f-a798-2072f63c243c" (UID: "885dd171-716d-4d0f-a798-2072f63c243c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:35.992810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992739    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:35.992810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.992661    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-model-cache" pod "885dd171-716d-4d0f-a798-2072f63c243c" (UID: "885dd171-716d-4d0f-a798-2072f63c243c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:35.994857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.994833    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885dd171-716d-4d0f-a798-2072f63c243c-kube-api-access-hg92x" pod "885dd171-716d-4d0f-a798-2072f63c243c" (UID: "885dd171-716d-4d0f-a798-2072f63c243c"). InnerVolumeSpecName "kube-api-access-hg92x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:35.995029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.994953    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-dshm" pod "885dd171-716d-4d0f-a798-2072f63c243c" (UID: "885dd171-716d-4d0f-a798-2072f63c243c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:35.995127 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:35.995040    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/885dd171-716d-4d0f-a798-2072f63c243c-tls-certs" pod "885dd171-716d-4d0f-a798-2072f63c243c" (UID: "885dd171-716d-4d0f-a798-2072f63c243c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:36.065656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.065613    5282 generic.go:356] "Generic (PLEG): container finished" podID="885dd171-716d-4d0f-a798-2072f63c243c" containerID="b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d" exitCode=0
Nov 03 20:27:36.065656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.065656    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"
Nov 03 20:27:36.065994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.065688    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" event={"ID":"885dd171-716d-4d0f-a798-2072f63c243c","Type":"ContainerDied","Data":"b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d"}
Nov 03 20:27:36.065994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.065730    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5" event={"ID":"885dd171-716d-4d0f-a798-2072f63c243c","Type":"ContainerDied","Data":"3292e8fe6958c6ec30b45b994d0216760a04112ae85818a6d1eda4771d69ee1b"}
Nov 03 20:27:36.065994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.065748    5282 scope.go:122] "RemoveContainer" containerID="b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d"
Nov 03 20:27:36.076139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.076121    5282 scope.go:122] "RemoveContainer" containerID="b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d"
Nov 03 20:27:36.076534 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:36.076513    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d\": container with ID starting with b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d not found: ID does not exist" containerID="b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d"
Nov 03 20:27:36.076606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.076547    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d"} err="failed to get container status \"b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d\": rpc error: code = NotFound desc = could not find container \"b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d\": container with ID starting with b22c4f3507f3e78b4973fe9ebc5f6c7cba27188064d9c8b1459de47d46e47c0d not found: ID does not exist"
Nov 03 20:27:36.094316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.094290    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.094316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.094314    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/885dd171-716d-4d0f-a798-2072f63c243c-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.094484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.094325    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg92x\" (UniqueName: \"kubernetes.io/projected/885dd171-716d-4d0f-a798-2072f63c243c-kube-api-access-hg92x\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.094484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.094337    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.094484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.094346    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/885dd171-716d-4d0f-a798-2072f63c243c-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.101439 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.101412    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"]
Nov 03 20:27:36.110126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:36.110099    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-11a5b0/distinct-2-w2-11a5b0-kserve-79bc866886-xdfm5"]
Nov 03 20:27:37.453075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:37.453037    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="885dd171-716d-4d0f-a798-2072f63c243c" path="/var/lib/kubelet/pods/885dd171-716d-4d0f-a798-2072f63c243c/volumes"
Nov 03 20:27:38.986680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:38.986641    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:27:41.647651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:41.647616    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"]
Nov 03 20:27:41.648082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:41.647840    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m" podUID="99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" containerName="maas-api" containerID="cri-o://1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003" gracePeriod=30
Nov 03 20:27:41.841667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:41.841626    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:41.895683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:41.895652    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:27:41.976151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:41.976052    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:42.051669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.051629    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-metrics-certs\") pod \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") "
Nov 03 20:27:42.051891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.051682    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-openshift-service-ca\") pod \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") "
Nov 03 20:27:42.051891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.051701    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-maas-api-tls\") pod \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") "
Nov 03 20:27:42.051891 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.051868    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-kube-api-access-btsq6\" (UniqueName: \"kubernetes.io/projected/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-kube-api-access-btsq6\") pod \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\" (UID: \"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81\") "
Nov 03 20:27:42.055570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.052505    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-openshift-service-ca" pod "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" (UID: "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:42.058093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.058042    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-metrics-certs" pod "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" (UID: "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:42.058293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.058233    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-maas-api-tls" pod "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" (UID: "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:27:42.060448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.060416    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-kube-api-access-btsq6" pod "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" (UID: "99f2c16b-4e2a-4530-a8b8-ee032f0d4d81"). InnerVolumeSpecName "kube-api-access-btsq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:42.096708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.096673    5282 generic.go:356] "Generic (PLEG): container finished" podID="99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" containerID="1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003" exitCode=0
Nov 03 20:27:42.096708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.096713    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m" event={"ID":"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81","Type":"ContainerDied","Data":"1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003"}
Nov 03 20:27:42.096953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.096722    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"
Nov 03 20:27:42.096953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.096752    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m" event={"ID":"99f2c16b-4e2a-4530-a8b8-ee032f0d4d81","Type":"ContainerDied","Data":"d9ca60ef198a4e31b8fd8ff478e5b7357afb99b2491ca0a7172995ae35bc960e"}
Nov 03 20:27:42.096953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.096774    5282 scope.go:122] "RemoveContainer" containerID="1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003"
Nov 03 20:27:42.109399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.108868    5282 scope.go:122] "RemoveContainer" containerID="1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003"
Nov 03 20:27:42.109399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.109072    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:42.109399 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:42.109350    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003\": container with ID starting with 1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003 not found: ID does not exist" containerID="1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003"
Nov 03 20:27:42.109399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.109383    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003"} err="failed to get container status \"1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003\": rpc error: code = NotFound desc = could not find container \"1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003\": container with ID starting with 1d1dc16bfc3d2c856d3b990025ee4d825d2674b1f5a26ee421cead28a99d7003 not found: ID does not exist"
Nov 03 20:27:42.128299 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.128255    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"]
Nov 03 20:27:42.134287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.134255    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-11a5b0-5fcccdb95c-nsf2m"]
Nov 03 20:27:42.153010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.152981    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:42.153010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.153009    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:42.153235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.153019    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:42.153235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.153029    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsq6\" (UniqueName: \"kubernetes.io/projected/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81-kube-api-access-btsq6\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:42.565981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.565947    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:42.864956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:42.864864    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:43.453532 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:43.453498    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" path="/var/lib/kubelet/pods/99f2c16b-4e2a-4530-a8b8-ee032f0d4d81/volumes"
Nov 03 20:27:44.860877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:44.860836    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:27:48.500362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:48.500331    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:48.741601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:48.741565    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:48.804224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:48.804121    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"]
Nov 03 20:27:48.804435 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:48.804410    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy" containerID="cri-o://51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3" gracePeriod=30
Nov 03 20:27:48.864918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:48.864878    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:53.103395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.103354    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-sl8rg"]
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.103917    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0ecdc4e5-5885-4c49-85b4-a58ed83fd584" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.103936    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecdc4e5-5885-4c49-85b4-a58ed83fd584" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.103956    5282 cpu_manager.go:414] "containerMap: removing container" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.103966    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.103979    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.103988    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104000    5282 cpu_manager.go:414] "containerMap: removing container" podUID="bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104008    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104022    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104031    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104044    5282 cpu_manager.go:414] "containerMap: removing container" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104052    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104061    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104068    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104082    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a7b2bd90-9bb5-469d-a8cb-194f786a1469" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104089    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b2bd90-9bb5-469d-a8cb-194f786a1469" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104102    5282 cpu_manager.go:414] "containerMap: removing container" podUID="529c2f87-65fc-44d1-94dd-c9f52ff20342" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104109    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529c2f87-65fc-44d1-94dd-c9f52ff20342" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104129    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f866abde-bff2-407c-918e-fa4aa493084c" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104137    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f866abde-bff2-407c-918e-fa4aa493084c" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104149    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104158    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104171    5282 cpu_manager.go:414] "containerMap: removing container" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104178    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104188    5282 cpu_manager.go:414] "containerMap: removing container" podUID="abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104226    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104246    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104254    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104265    5282 cpu_manager.go:414] "containerMap: removing container" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104272    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104285    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104292    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104304    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104312    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104322    5282 cpu_manager.go:414] "containerMap: removing container" podUID="03dece0c-cf3c-4d61-833d-a7a8a3531666" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104329    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03dece0c-cf3c-4d61-833d-a7a8a3531666" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104338    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104345    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104359    5282 cpu_manager.go:414] "containerMap: removing container" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104367    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104381    5282 cpu_manager.go:414] "containerMap: removing container" podUID="6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104388    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104400    5282 cpu_manager.go:414] "containerMap: removing container" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104407    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104416    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ca057d53-559b-4e56-9635-24ef10b0b325" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104423    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca057d53-559b-4e56-9635-24ef10b0b325" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104433    5282 cpu_manager.go:414] "containerMap: removing container" podUID="76eaf966-caf0-4394-9f02-488f93755253" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104439    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76eaf966-caf0-4394-9f02-488f93755253" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104453    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104460    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104472    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104481    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104491    5282 cpu_manager.go:414] "containerMap: removing container" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104498    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104508    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104515    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104527    5282 cpu_manager.go:414] "containerMap: removing container" podUID="777432e5-8d85-41bc-b1d3-851ee8926850" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104535    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777432e5-8d85-41bc-b1d3-851ee8926850" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104544    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104551    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104563    5282 cpu_manager.go:414] "containerMap: removing container" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104570    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104579    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104586    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104600    5282 cpu_manager.go:414] "containerMap: removing container" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104606    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104616    5282 cpu_manager.go:414] "containerMap: removing container" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104623    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104634    5282 cpu_manager.go:414] "containerMap: removing container" podUID="68cfe518-7312-4bbb-8700-f8a165a77251" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104641    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cfe518-7312-4bbb-8700-f8a165a77251" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104658    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0d433013-5364-4ebe-96d9-08e53103e387" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104665    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d433013-5364-4ebe-96d9-08e53103e387" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104676    5282 cpu_manager.go:414] "containerMap: removing container" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104682    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104693    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5ce92ce4-1f60-438c-8741-0c2238548df4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104700    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce92ce4-1f60-438c-8741-0c2238548df4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104711    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104718    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104730    5282 cpu_manager.go:414] "containerMap: removing container" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104738    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104747    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104754    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104764    5282 cpu_manager.go:414] "containerMap: removing container" podUID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104772    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104786    5282 cpu_manager.go:414] "containerMap: removing container" podUID="99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104795    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104810    5282 cpu_manager.go:414] "containerMap: removing container" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104817    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104923    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca057d53-559b-4e56-9635-24ef10b0b325" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104937    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="76eaf966-caf0-4394-9f02-488f93755253" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104947    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f873dc7e-faaa-4b5d-8b32-298726b5aef8" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104955    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e6368c-b747-4898-bed9-6d4e60dda094" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104964    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ebd8ea9-701c-4476-9380-9856f565e819" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104973    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a6d7c7-3fe1-45dd-a4ff-2b66c0547cac" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104986    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="777432e5-8d85-41bc-b1d3-851ee8926850" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.104997    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc9e6e9-6e8e-42c9-bb88-7d4f7fd8a2fa" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105005    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ec540a-467b-4a58-962a-4b804ebfb2ca" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105015    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7c63cf-607d-467c-9239-9ed4e1e3a6ba" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105024    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40ae433-7695-44d1-ba58-0614ce2aa2f5" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105034    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6e5cd9-7b07-4b92-881b-81deaf1524c2" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105043    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cfe518-7312-4bbb-8700-f8a165a77251" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105054    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d433013-5364-4ebe-96d9-08e53103e387" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105062    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b82d548-716d-4a3e-98f1-0b3c8e4c91b0" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105071    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce92ce4-1f60-438c-8741-0c2238548df4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105081    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105091    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105099    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08629be-9d3a-4e49-8935-e9f089de7ce9" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105108    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4e8061-b105-45fc-8a9e-a35605b4bcb7" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105117    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b253246c-0adb-4d4d-b6db-4bfa0c3993f4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105126    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e978959-92f5-4c2a-afc3-91ae95b77dd9" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105134    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f2c16b-4e2a-4530-a8b8-ee032f0d4d81" containerName="maas-api"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105144    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d36fc54-8a2c-45c0-9b1a-b7d441c1d947" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105152    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecdc4e5-5885-4c49-85b4-a58ed83fd584" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105162    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105170    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8e93e9-5642-420b-96df-068346841dd4" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105178    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2ceb5a-f3de-40c5-91ba-eed7d0c374ac" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105187    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105215    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105224    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47726d4-a756-464a-9fff-465b9ab355b0" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105233    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b2bd90-9bb5-469d-a8cb-194f786a1469" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105241    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="529c2f87-65fc-44d1-94dd-c9f52ff20342" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105252    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f866abde-bff2-407c-918e-fa4aa493084c" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105261    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4f9c0c-1c61-42e5-a0e1-f467d0d6ccc8" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105270    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="885dd171-716d-4d0f-a798-2072f63c243c" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105279    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd7dabe-9b92-4f7d-be7a-0f9e3b4d22c3" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105289    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f777c24d-c5bf-4e9e-a65f-eef2aa170470" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105299    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7652f6b-b3d5-4aa5-a49e-ab25fc8f65b3" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105309    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f48c0353-24d5-43a4-ba99-ee8d666ad6a2" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105318    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3416cbfd-53c7-494a-81e5-ec52c63d39d6" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105328    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="03dece0c-cf3c-4d61-833d-a7a8a3531666" containerName="payload-pre-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105338    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0213ef7-a48f-4746-8c08-57b1a3961c1d" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105348    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d5d678-8552-4d54-a140-210d27813c3d" containerName="main"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105357    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7ba8dc-4ae4-443e-8d7e-c65c98362bd4" containerName="payload-processing"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105366    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34ad4b7-7a6e-4eb7-a6b6-d5eaa0ef89e8" containerName="istio-proxy"
Nov 03 20:27:53.105933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.105375    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbfc761-e296-4114-b4e3-da2df1301934" containerName="istio-proxy"
Nov 03 20:27:53.110820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.108678    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.127565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.127533    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-sl8rg"]
Nov 03 20:27:53.130649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.130623    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:27:53.132540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.130947    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:27:53.132699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.131042    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dlrg\""
Nov 03 20:27:53.132779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.131741    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:27:53.148885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.148852    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k4g\" (UniqueName: \"kubernetes.io/projected/6ae6c48f-d76f-4444-a579-50f110e656d8-kube-api-access-n4k4g\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.149059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.148931    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-metrics-certs\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.149059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.149049    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-webhook-cert\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.250480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.250440    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-webhook-cert\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.250674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.250548    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k4g\" (UniqueName: \"kubernetes.io/projected/6ae6c48f-d76f-4444-a579-50f110e656d8-kube-api-access-n4k4g\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.250674 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.250606    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-metrics-certs\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.253095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.253062    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-metrics-certs\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.253095 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.253080    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-webhook-cert\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.258474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.258447    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k4g\" (UniqueName: \"kubernetes.io/projected/6ae6c48f-d76f-4444-a579-50f110e656d8-kube-api-access-n4k4g\") pod \"maas-controller-65f5dc4b79-sl8rg\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") " pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.440442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.440345    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:53.718859 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.718743    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-sl8rg"]
Nov 03 20:27:53.725271 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:53.723795    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:27:54.102724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.101451    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:27:54.159152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159117    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-kube-api-access-856bg\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-kube-api-access-856bg\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159180    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-certs\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159224    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-data\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159322    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-token\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-token\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159375    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-socket\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159417    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-envoy\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159445    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-podinfo\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159477    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266b78d6-34f2-424e-9ac0-3b9eb3684499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b78d6-34f2-424e-9ac0-3b9eb3684499-istiod-ca-cert\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159496    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-certs" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.159657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159516    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-credential-socket\") pod \"266b78d6-34f2-424e-9ac0-3b9eb3684499\" (UID: \"266b78d6-34f2-424e-9ac0-3b9eb3684499\") "
Nov 03 20:27:54.160002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159832    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.160002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.159877    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-socket" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.160077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.160023    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-credential-socket" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.160077 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.160066    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-data" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.161277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.160907    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg" event={"ID":"6ae6c48f-d76f-4444-a579-50f110e656d8","Type":"ContainerStarted","Data":"01a77c227ff7ec6ff52478736071e5581dab9b62ec4e0b39d98ee50ffa2fa7b2"}
Nov 03 20:27:54.161277 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.160926    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266b78d6-34f2-424e-9ac0-3b9eb3684499-istiod-ca-cert" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:54.161955 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.161929    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-kube-api-access-856bg" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "kube-api-access-856bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:54.162310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.162276    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-token" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:54.162438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.162410    5282 generic.go:356] "Generic (PLEG): container finished" podID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerID="51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3" exitCode=0
Nov 03 20:27:54.162512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.162443    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" event={"ID":"266b78d6-34f2-424e-9ac0-3b9eb3684499","Type":"ContainerDied","Data":"51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3"}
Nov 03 20:27:54.162512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.162462    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"
Nov 03 20:27:54.162512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.162495    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl" event={"ID":"266b78d6-34f2-424e-9ac0-3b9eb3684499","Type":"ContainerDied","Data":"32fb875f813b3d4e29d9a168153d0e45585bdca1a516c6f0b948a1653e0daa18"}
Nov 03 20:27:54.162661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.162526    5282 scope.go:122] "RemoveContainer" containerID="51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3"
Nov 03 20:27:54.162661 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.162420    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-envoy" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:54.163827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.163804    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-podinfo" pod "266b78d6-34f2-424e-9ac0-3b9eb3684499" (UID: "266b78d6-34f2-424e-9ac0-3b9eb3684499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:54.177782 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.177752    5282 scope.go:122] "RemoveContainer" containerID="51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3"
Nov 03 20:27:54.178460 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:27:54.178379    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3\": container with ID starting with 51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3 not found: ID does not exist" containerID="51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3"
Nov 03 20:27:54.178460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.178422    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3"} err="failed to get container status \"51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3\": rpc error: code = NotFound desc = could not find container \"51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3\": container with ID starting with 51e905885fb6193469a348b6e12073f26ad1f9fe7f4ee81cfad1ec9360d635d3 not found: ID does not exist"
Nov 03 20:27:54.261141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261110    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.261141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261139    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.261342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261154    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.261342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261166    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266b78d6-34f2-424e-9ac0-3b9eb3684499-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.261342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261175    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.261342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261184    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-856bg\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-kube-api-access-856bg\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.261342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261209    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.261342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.261221    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266b78d6-34f2-424e-9ac0-3b9eb3684499-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:54.494222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.488750    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"]
Nov 03 20:27:54.509285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:54.498436    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-11a5b0-openshift-default-8569845457-frxxl"]
Nov 03 20:27:55.453813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:55.453778    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" path="/var/lib/kubelet/pods/266b78d6-34f2-424e-9ac0-3b9eb3684499/volumes"
Nov 03 20:27:56.173947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:56.173855    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg" event={"ID":"6ae6c48f-d76f-4444-a579-50f110e656d8","Type":"ContainerStarted","Data":"a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020"}
Nov 03 20:27:56.173947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:56.173915    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:27:56.193991 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:56.193937    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg" podStartSLOduration=1.143430634 podStartE2EDuration="3.193923047s" totalImagesPullingTime="2.050492413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:56.192110156 +0000 UTC m=+2868.861311643" watchObservedRunningTime="2026-11-03 20:27:56.193923047 +0000 UTC m=+2868.863124516"
Nov 03 20:27:57.922121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:57.922087    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:58.062732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:58.062693    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:27:58.539151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:27:58.539112    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:01.613846 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:28:01.613809    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:28:04.337977 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:04.337937    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:07.181456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:07.181418    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:28:11.114248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:11.114215    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:11.230482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:11.230446    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:17.809526 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809487    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"]
Nov 03 20:28:17.809922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809869    5282 cpu_manager.go:414] "containerMap: removing container" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy"
Nov 03 20:28:17.809922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809880    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347cb059-324b-4249-b74b-2a18fd8d045a" containerName="istio-proxy"
Nov 03 20:28:17.809922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809889    5282 cpu_manager.go:414] "containerMap: removing container" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy"
Nov 03 20:28:17.809922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809894    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy"
Nov 03 20:28:17.809922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809902    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy"
Nov 03 20:28:17.809922 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809908    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a030b91-7598-43b4-8554-ba348887cf30" containerName="istio-proxy"
Nov 03 20:28:17.810169 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.809993    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="266b78d6-34f2-424e-9ac0-3b9eb3684499" containerName="istio-proxy"
Nov 03 20:28:17.815104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.815081    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.826258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.826227    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"]
Nov 03 20:28:17.839713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.839681    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.841022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.840980    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"]
Nov 03 20:28:17.843720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.843683    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"]
Nov 03 20:28:17.880302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880268    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880324    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880510 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880394    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880520    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880638 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880567    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880641    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwd2\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-kube-api-access-szwd2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880668    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880726    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c607312e-afa0-4bcf-8885-c2862a324ed8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.880864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.880750    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.981876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.981835    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.981876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.981878    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pkx\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-kube-api-access-z4pkx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.982172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.981901    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.982172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982035    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c607312e-afa0-4bcf-8885-c2862a324ed8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982091    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.982172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982127    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982172 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982163    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982232    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.982420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982350    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982420 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982369    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.982585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982421    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982479    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982507    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982585 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982557    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.982799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982620    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982649    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.982799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982671    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982679    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.982799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982752    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c607312e-afa0-4bcf-8885-c2862a324ed8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.983047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982802    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.983047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982799    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc1995-f631-409c-afd1-a3cd057a51a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.983047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982870    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwd2\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-kube-api-access-szwd2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.983047 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.982916    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:17.984832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.984813    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.984907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.984868    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.991335 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.991303    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:17.991612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:17.991589    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwd2\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-kube-api-access-szwd2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-45b2b\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:18.084402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084308    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084355    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc1995-f631-409c-afd1-a3cd057a51a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084383    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084412    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pkx\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-kube-api-access-z4pkx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084440    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084487    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084523    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084554    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084609    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084850    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.084987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.084938    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.085091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.085028    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.085154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.085099    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc1995-f631-409c-afd1-a3cd057a51a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.085227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.085192    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.086885 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.086857    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.087375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.087348    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.092123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.092093    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.092266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.092156    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pkx\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-kube-api-access-z4pkx\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cwz84\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.141280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.141246    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:18.160637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.160606    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:18.422812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.422781    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"]
Nov 03 20:28:18.424962 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:28:18.424927    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc607312e_afa0_4bcf_8885_c2862a324ed8.slice/crio-8f606e048a98e4c5dc62e327b7ed9efb4076db13d33088d6331a40d733627858 WatchSource:0}: Error finding container 8f606e048a98e4c5dc62e327b7ed9efb4076db13d33088d6331a40d733627858: Status 404 returned error can't find the container with id 8f606e048a98e4c5dc62e327b7ed9efb4076db13d33088d6331a40d733627858
Nov 03 20:28:18.427072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.427038    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:28:18.427130 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.427111    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:28:18.427167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.427143    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:28:18.452755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.452729    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"]
Nov 03 20:28:18.463471 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:28:18.463440    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cc1995_f631_409c_afd1_a3cd057a51a2.slice/crio-d584c79f13bf611f36a09e90de3493d55053d6fa54909493f37b90ef1fcb0501 WatchSource:0}: Error finding container d584c79f13bf611f36a09e90de3493d55053d6fa54909493f37b90ef1fcb0501: Status 404 returned error can't find the container with id d584c79f13bf611f36a09e90de3493d55053d6fa54909493f37b90ef1fcb0501
Nov 03 20:28:18.465965 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.465922    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:28:18.466053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.466023    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:28:18.466099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:18.466073    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:28:19.325347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:19.325252    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" event={"ID":"c607312e-afa0-4bcf-8885-c2862a324ed8","Type":"ContainerStarted","Data":"8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee"}
Nov 03 20:28:19.325938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:19.325916    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" event={"ID":"c607312e-afa0-4bcf-8885-c2862a324ed8","Type":"ContainerStarted","Data":"8f606e048a98e4c5dc62e327b7ed9efb4076db13d33088d6331a40d733627858"}
Nov 03 20:28:19.327604 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:19.327575    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" event={"ID":"b5cc1995-f631-409c-afd1-a3cd057a51a2","Type":"ContainerStarted","Data":"c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823"}
Nov 03 20:28:19.327702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:19.327631    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" event={"ID":"b5cc1995-f631-409c-afd1-a3cd057a51a2","Type":"ContainerStarted","Data":"d584c79f13bf611f36a09e90de3493d55053d6fa54909493f37b90ef1fcb0501"}
Nov 03 20:28:19.344235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:19.344155    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podStartSLOduration=2.344138945 podStartE2EDuration="2.344138945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:19.343650517 +0000 UTC m=+2892.012851983" watchObservedRunningTime="2026-11-03 20:28:19.344138945 +0000 UTC m=+2892.013340413"
Nov 03 20:28:19.363944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:19.362986    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podStartSLOduration=2.362965854 podStartE2EDuration="2.362965854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:19.361603259 +0000 UTC m=+2892.030804733" watchObservedRunningTime="2026-11-03 20:28:19.362965854 +0000 UTC m=+2892.032167327"
Nov 03 20:28:20.141762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:20.141730    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:20.142914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:20.142889    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-45b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:28:20.143001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:20.142934    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 20:28:20.161262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:20.161234    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:20.162729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:20.162690    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:28:20.162812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:20.162755    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 20:28:21.142362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:21.142314    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-45b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.142861 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:21.142386    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 20:28:21.161897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:21.161864    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.162093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:21.162069    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 20:28:22.101221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:22.093563    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:22.141729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:22.141685    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-45b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.142032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:22.141988    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 20:28:22.161054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:22.161018    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.161573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:22.161068    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 20:28:23.141643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:23.141603    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-45b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.141837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:23.141661    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 20:28:23.161371 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:23.161329    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.161775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:23.161430    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 20:28:24.143017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:24.141773    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-45b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.143017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:24.141829    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 20:28:24.161256 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:24.161214    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.161589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:24.161538    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 20:28:25.142054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:25.142012    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-45b2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.142291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:25.142078    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 20:28:25.161580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:25.161545    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.161754 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:25.161641    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 20:28:25.671346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:25.671309    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:25.745221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:25.745168    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-sl8rg"]
Nov 03 20:28:25.745477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:25.745438    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg" podUID="6ae6c48f-d76f-4444-a579-50f110e656d8" containerName="manager" containerID="cri-o://a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020" gracePeriod=30
Nov 03 20:28:26.016412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.016378    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:28:26.134362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.134306    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae6c48f-d76f-4444-a579-50f110e656d8-kube-api-access-n4k4g\" (UniqueName: \"kubernetes.io/projected/6ae6c48f-d76f-4444-a579-50f110e656d8-kube-api-access-n4k4g\") pod \"6ae6c48f-d76f-4444-a579-50f110e656d8\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") "
Nov 03 20:28:26.134588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.134392    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-metrics-certs\") pod \"6ae6c48f-d76f-4444-a579-50f110e656d8\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") "
Nov 03 20:28:26.134588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.134511    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-webhook-cert\") pod \"6ae6c48f-d76f-4444-a579-50f110e656d8\" (UID: \"6ae6c48f-d76f-4444-a579-50f110e656d8\") "
Nov 03 20:28:26.137902 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.137849    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae6c48f-d76f-4444-a579-50f110e656d8-kube-api-access-n4k4g" pod "6ae6c48f-d76f-4444-a579-50f110e656d8" (UID: "6ae6c48f-d76f-4444-a579-50f110e656d8"). InnerVolumeSpecName "kube-api-access-n4k4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:26.146484 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.146286    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-webhook-cert" pod "6ae6c48f-d76f-4444-a579-50f110e656d8" (UID: "6ae6c48f-d76f-4444-a579-50f110e656d8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:26.151229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.149333    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-metrics-certs" pod "6ae6c48f-d76f-4444-a579-50f110e656d8" (UID: "6ae6c48f-d76f-4444-a579-50f110e656d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:28:26.236066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.235992    5282 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-webhook-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:26.236066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.236058    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k4g\" (UniqueName: \"kubernetes.io/projected/6ae6c48f-d76f-4444-a579-50f110e656d8-kube-api-access-n4k4g\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:26.236066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.236076    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ae6c48f-d76f-4444-a579-50f110e656d8-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:26.369548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.369503    5282 generic.go:356] "Generic (PLEG): container finished" podID="6ae6c48f-d76f-4444-a579-50f110e656d8" containerID="a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020" exitCode=0
Nov 03 20:28:26.369787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.369568    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg"
Nov 03 20:28:26.369787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.369601    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg" event={"ID":"6ae6c48f-d76f-4444-a579-50f110e656d8","Type":"ContainerDied","Data":"a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020"}
Nov 03 20:28:26.369787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.369668    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-sl8rg" event={"ID":"6ae6c48f-d76f-4444-a579-50f110e656d8","Type":"ContainerDied","Data":"01a77c227ff7ec6ff52478736071e5581dab9b62ec4e0b39d98ee50ffa2fa7b2"}
Nov 03 20:28:26.369787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.369702    5282 scope.go:122] "RemoveContainer" containerID="a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020"
Nov 03 20:28:26.382400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.382375    5282 scope.go:122] "RemoveContainer" containerID="a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020"
Nov 03 20:28:26.382811 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:28:26.382781    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020\": container with ID starting with a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020 not found: ID does not exist" containerID="a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020"
Nov 03 20:28:26.382908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.382828    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020"} err="failed to get container status \"a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020\": rpc error: code = NotFound desc = could not find container \"a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020\": container with ID starting with a6fc5651f8c7c3ac7e7754a3a41be07846adfe815cee7f4ff0c7e13de92c0020 not found: ID does not exist"
Nov 03 20:28:26.396460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.396421    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-sl8rg"]
Nov 03 20:28:26.400399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.400368    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-sl8rg"]
Nov 03 20:28:26.838888 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.838852    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:26.839593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.839571    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:26.842119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.842096    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:28:26.963002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.962967    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:26.963501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.963472    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:26.964413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:26.964393    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:28:27.454102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.454067    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae6c48f-d76f-4444-a579-50f110e656d8" path="/var/lib/kubelet/pods/6ae6c48f-d76f-4444-a579-50f110e656d8/volumes"
Nov 03 20:28:27.489876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.489842    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:28:27.584615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.584582    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ptlhp"]
Nov 03 20:28:27.584987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.584974    5282 cpu_manager.go:414] "containerMap: removing container" podUID="6ae6c48f-d76f-4444-a579-50f110e656d8" containerName="manager"
Nov 03 20:28:27.585031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.584987    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae6c48f-d76f-4444-a579-50f110e656d8" containerName="manager"
Nov 03 20:28:27.585069 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.585051    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae6c48f-d76f-4444-a579-50f110e656d8" containerName="manager"
Nov 03 20:28:27.589702 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.589675    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.593190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.593163    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:28:27.598258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.598229    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ptlhp"]
Nov 03 20:28:27.604737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.604705    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:28:27.604737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.604730    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dlrg\""
Nov 03 20:28:27.604925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.604783    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:28:27.752303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.752264    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3f2c57-f26a-48ce-bb62-ce619f8d622f-metrics-certs\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.752463 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.752337    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3f2c57-f26a-48ce-bb62-ce619f8d622f-webhook-cert\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.752531 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.752511    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tkd\" (UniqueName: \"kubernetes.io/projected/db3f2c57-f26a-48ce-bb62-ce619f8d622f-kube-api-access-q9tkd\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.853622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.853579    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tkd\" (UniqueName: \"kubernetes.io/projected/db3f2c57-f26a-48ce-bb62-ce619f8d622f-kube-api-access-q9tkd\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.853808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.853634    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3f2c57-f26a-48ce-bb62-ce619f8d622f-metrics-certs\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.853808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.853680    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3f2c57-f26a-48ce-bb62-ce619f8d622f-webhook-cert\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.856296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.856265    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3f2c57-f26a-48ce-bb62-ce619f8d622f-metrics-certs\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.856413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.856325    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/db3f2c57-f26a-48ce-bb62-ce619f8d622f-webhook-cert\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.862045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.862027    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tkd\" (UniqueName: \"kubernetes.io/projected/db3f2c57-f26a-48ce-bb62-ce619f8d622f-kube-api-access-q9tkd\") pod \"maas-controller-65f5dc4b79-ptlhp\" (UID: \"db3f2c57-f26a-48ce-bb62-ce619f8d622f\") " pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:27.904315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:27.904259    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:28.177091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:28.177065    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5dc4b79-ptlhp"]
Nov 03 20:28:28.179037 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:28:28.179007    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3f2c57_f26a_48ce_bb62_ce619f8d622f.slice/crio-7f37d87cb337bc60476df0f7ebfeff799ddb4c607cb7604ed295e69725c40b85 WatchSource:0}: Error finding container 7f37d87cb337bc60476df0f7ebfeff799ddb4c607cb7604ed295e69725c40b85: Status 404 returned error can't find the container with id 7f37d87cb337bc60476df0f7ebfeff799ddb4c607cb7604ed295e69725c40b85
Nov 03 20:28:28.381011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:28.380968    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-ptlhp" event={"ID":"db3f2c57-f26a-48ce-bb62-ce619f8d622f","Type":"ContainerStarted","Data":"7f37d87cb337bc60476df0f7ebfeff799ddb4c607cb7604ed295e69725c40b85"}
Nov 03 20:28:29.387321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:29.387276    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5dc4b79-ptlhp" event={"ID":"db3f2c57-f26a-48ce-bb62-ce619f8d622f","Type":"ContainerStarted","Data":"35c860303a30497e5b43c5de6e2fc4f9e45c7760c266b94f3505d7010e208f65"}
Nov 03 20:28:29.387850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:29.387405    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:28:29.404240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:29.404163    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65f5dc4b79-ptlhp" podStartSLOduration=1.982452189 podStartE2EDuration="2.404150858s" totalImagesPullingTime="421.698669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:29.403253268 +0000 UTC m=+2902.072454739" watchObservedRunningTime="2026-11-03 20:28:29.404150858 +0000 UTC m=+2902.073352328"
Nov 03 20:28:31.613332 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:28:31.613295    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:28:40.394022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:28:40.393992    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f5dc4b79-ptlhp"
Nov 03 20:29:01.614179 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:29:01.614134    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:29:02.202933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:29:02.202891    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:29:20.385512 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:29:20.385472    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:29:22.135102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:29:22.135031    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:29:25.457419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:29:25.457390    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:29:31.613515 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:29:31.613469    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:29:39.154242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:29:39.154206    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:29:51.875770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:29:51.875733    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:30:01.614253 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:30:01.614210    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:30:02.480449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:30:02.480413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:16.555675 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:30:16.555635    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:30:22.139132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:30:22.139089    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:30:26.652381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:30:26.652343    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:30:31.361573 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:30:31.361531    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:30:31.617005 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:30:31.616908    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:30:47.179725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:30:47.179686    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:30:50.938231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:30:50.932804    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:01.615020 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:31:01.614980    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:31:05.446452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:05.446417    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:10.548239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:10.547258    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:11.351449 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:11.351413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:22.228215 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:22.228110    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:31:24.062951 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:24.062916    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:27.741489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:27.741449    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:28.015745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:28.015310    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:31:28.015745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:28.015383    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:31.612994 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:31:31.612953    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:31:41.792717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:41.792673    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:52.562505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:52.562466    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:56.824928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:56.824877    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:31:57.891365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:57.891321    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-45b2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:31:57.892065 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:57.892028    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:58.008221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:58.007321    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cwz84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:31:58.008221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:31:58.007397    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:32:01.614129 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:32:01.614091    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:32:09.281815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:09.281776    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:09.401564 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:09.401532    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:11.079216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:11.079166    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:32:14.987582 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:14.987541    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:32:20.855283 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:20.855243    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:32:22.359223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:22.358367    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:32:23.420358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:23.419820    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:32:23.420358 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:23.419889    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:32:23.471233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:23.468309    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:32:23.471233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:23.468376    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:32:31.612656 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:32:31.612622    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:32:38.981996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:38.981959    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:32:49.174766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:32:49.174688    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:33:01.612744 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:33:01.612707    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:33:23.695453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:33:23.695413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:33:31.613378 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:33:31.613340    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:34:01.613867 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:34:01.613825    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:34:22.450485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:34:22.450390    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:34:31.614487 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:34:31.614450    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:34:44.812833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:34:44.812799    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:01.613917 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:35:01.613874    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:35:22.501054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:35:22.501016    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:35:31.613953 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:35:31.613906    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:36:01.613836 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:36:01.613755    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:36:22.494246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:36:22.494184    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:36:31.614175 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:36:31.614137    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:37:01.614613 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:37:01.614574    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:37:11.292534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:37:11.292480    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:22.599490 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:37:22.599386    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:37:31.613990 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:37:31.613947    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:38:01.613241 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:38:01.613114    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:38:03.398844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.398803    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"]
Nov 03 20:38:03.399431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.399011    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" containerID="cri-o://93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a" gracePeriod=30
Nov 03 20:38:03.406286 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.406250    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"]
Nov 03 20:38:03.406565 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.406532    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy" containerID="cri-o://c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823" gracePeriod=30
Nov 03 20:38:03.407470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.407448    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"]
Nov 03 20:38:03.407809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.407761    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy" containerID="cri-o://8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee" gracePeriod=30
Nov 03 20:38:03.410606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.410579    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"]
Nov 03 20:38:03.410809 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:03.410783    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" containerID="cri-o://caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a" gracePeriod=30
Nov 03 20:38:06.868533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:06.868495    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:38:08.415569 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.415530    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2n8gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:38:08.416126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.416078    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:38:08.423724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.423579    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-7krc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:38:08.423724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.423661    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:38:08.679006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.678940    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:38:08.736916 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.736746    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 20:38:08.744626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.744601    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 20:38:08.748737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.748711    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:38:08.753970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.753947    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-envoy\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754006    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c607312e-afa0-4bcf-8885-c2862a324ed8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c607312e-afa0-4bcf-8885-c2862a324ed8-istiod-ca-cert\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754054    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-podinfo\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754102 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754089    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-certs\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754152    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-credential-socket\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754407    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-credential-socket" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.754527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754429    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-certs" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.754527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754453    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c607312e-afa0-4bcf-8885-c2862a324ed8-istiod-ca-cert" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.754527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754476    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-socket\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754623    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-socket" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.754756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754634    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-data\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754693    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-token\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-token\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754728    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-kube-api-access-szwd2\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-kube-api-access-szwd2\") pod \"c607312e-afa0-4bcf-8885-c2862a324ed8\" (UID: \"c607312e-afa0-4bcf-8885-c2862a324ed8\") "
Nov 03 20:38:08.754960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.754811    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-data" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.755083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.755065    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c607312e-afa0-4bcf-8885-c2862a324ed8-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.755148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.755090    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.755148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.755106    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.755148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.755121    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.755148 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.755134    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.756741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.756708    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-podinfo" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:08.756741 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.756718    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-envoy" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.756896 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.756838    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-kube-api-access-szwd2" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "kube-api-access-szwd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.757181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.757157    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-token" pod "c607312e-afa0-4bcf-8885-c2862a324ed8" (UID: "c607312e-afa0-4bcf-8885-c2862a324ed8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.855877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.855834    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-credential-socket\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.856101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.855900    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-podinfo\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.856101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.855932    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-data\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.856101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.855957    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-token\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.856101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.855985    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-socket\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.856101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856010    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-envoy\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.856101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856036    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-podinfo\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.856101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856071    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35226e43-d541-43f8-a888-edb3202ad0f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35226e43-d541-43f8-a888-edb3202ad0f6-istio-podinfo\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856105    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-kube-api-access-jdkcg\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-kube-api-access-jdkcg\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856147    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-certs\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856187    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-data\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856191    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-credential-socket" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856255    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-socket\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856289    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-certs\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856317    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-istio-token\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-istio-token\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856291    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-data" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856344    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-socket\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856349    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-socket" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856389    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5cc1995-f631-409c-afd1-a3cd057a51a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc1995-f631-409c-afd1-a3cd057a51a2-istiod-ca-cert\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856421    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-credential-socket\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856455    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-certs\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856500    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35226e43-d541-43f8-a888-edb3202ad0f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35226e43-d541-43f8-a888-edb3202ad0f6-istiod-ca-cert\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.856542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856529    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istiod-ca-cert\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856552    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-socket" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856556    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-token\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-token\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856609    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-certs" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856617    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-kube-api-access-z4pkx\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-kube-api-access-z4pkx\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856675    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-credential-socket\") pod \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\" (UID: \"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856730    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-envoy\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856782    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-data\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856811    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-kube-api-access-29w8n\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-kube-api-access-29w8n\") pod \"35226e43-d541-43f8-a888-edb3202ad0f6\" (UID: \"35226e43-d541-43f8-a888-edb3202ad0f6\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.856858    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-envoy\") pod \"b5cc1995-f631-409c-afd1-a3cd057a51a2\" (UID: \"b5cc1995-f631-409c-afd1-a3cd057a51a2\") "
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857353    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857378    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857395    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857410    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857423    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857437    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857449    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857464    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwd2\" (UniqueName: \"kubernetes.io/projected/c607312e-afa0-4bcf-8885-c2862a324ed8-kube-api-access-szwd2\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.857548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857480    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c607312e-afa0-4bcf-8885-c2862a324ed8-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857847    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-data" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.857945    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-credential-socket" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858107    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istiod-ca-cert" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858566    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35226e43-d541-43f8-a888-edb3202ad0f6-istiod-ca-cert" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858563    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-certs" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858619    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-credential-socket" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858634    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-certs" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858665    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-socket" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858758    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-data" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.859370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.858870    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cc1995-f631-409c-afd1-a3cd057a51a2-istiod-ca-cert" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:08.860025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.859528    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-podinfo" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:08.860170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.860146    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-podinfo" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:08.860296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.860160    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-token" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.860448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.860344    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-token" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.860448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.860365    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-kube-api-access-z4pkx" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "kube-api-access-z4pkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.860653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.860631    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35226e43-d541-43f8-a888-edb3202ad0f6-istio-podinfo" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:08.860968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.860943    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-envoy" pod "b5cc1995-f631-409c-afd1-a3cd057a51a2" (UID: "b5cc1995-f631-409c-afd1-a3cd057a51a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.861145 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.861115    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-istio-token" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.861235 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.861136    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-kube-api-access-29w8n" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "kube-api-access-29w8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.861390 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.861373    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-kube-api-access-jdkcg" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "kube-api-access-jdkcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:08.861757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.861742    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-envoy" pod "35226e43-d541-43f8-a888-edb3202ad0f6" (UID: "35226e43-d541-43f8-a888-edb3202ad0f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.861940 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.861923    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-envoy" pod "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" (UID: "59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:08.958411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958321    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958350    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958364    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958376    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958388    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35226e43-d541-43f8-a888-edb3202ad0f6-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958401    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdkcg\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-kube-api-access-jdkcg\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958417    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958431    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958442    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958454    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958465    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc1995-f631-409c-afd1-a3cd057a51a2-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958476    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958488    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958502    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35226e43-d541-43f8-a888-edb3202ad0f6-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958514    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958528    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958540    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4pkx\" (UniqueName: \"kubernetes.io/projected/b5cc1995-f631-409c-afd1-a3cd057a51a2-kube-api-access-z4pkx\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958553    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958564    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35226e43-d541-43f8-a888-edb3202ad0f6-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958577    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958588    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29w8n\" (UniqueName: \"kubernetes.io/projected/35226e43-d541-43f8-a888-edb3202ad0f6-kube-api-access-29w8n\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:08.958779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:08.958600    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc1995-f631-409c-afd1-a3cd057a51a2-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:09.415067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.415028    5282 generic.go:356] "Generic (PLEG): container finished" podID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerID="8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee" exitCode=0
Nov 03 20:38:09.415275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.415084    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"
Nov 03 20:38:09.415275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.415114    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" event={"ID":"c607312e-afa0-4bcf-8885-c2862a324ed8","Type":"ContainerDied","Data":"8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee"}
Nov 03 20:38:09.415275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.415187    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b" event={"ID":"c607312e-afa0-4bcf-8885-c2862a324ed8","Type":"ContainerDied","Data":"8f606e048a98e4c5dc62e327b7ed9efb4076db13d33088d6331a40d733627858"}
Nov 03 20:38:09.415275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.415226    5282 scope.go:122] "RemoveContainer" containerID="8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee"
Nov 03 20:38:09.416486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.416465    5282 generic.go:356] "Generic (PLEG): container finished" podID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerID="c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823" exitCode=0
Nov 03 20:38:09.416873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.416579    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"
Nov 03 20:38:09.416873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.416579    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" event={"ID":"b5cc1995-f631-409c-afd1-a3cd057a51a2","Type":"ContainerDied","Data":"c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823"}
Nov 03 20:38:09.416873 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.416704    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84" event={"ID":"b5cc1995-f631-409c-afd1-a3cd057a51a2","Type":"ContainerDied","Data":"d584c79f13bf611f36a09e90de3493d55053d6fa54909493f37b90ef1fcb0501"}
Nov 03 20:38:09.417953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.417928    5282 generic.go:356] "Generic (PLEG): container finished" podID="35226e43-d541-43f8-a888-edb3202ad0f6" containerID="93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a" exitCode=0
Nov 03 20:38:09.418066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.417971    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"
Nov 03 20:38:09.418066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.418004    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" event={"ID":"35226e43-d541-43f8-a888-edb3202ad0f6","Type":"ContainerDied","Data":"93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a"}
Nov 03 20:38:09.418066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.418045    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb" event={"ID":"35226e43-d541-43f8-a888-edb3202ad0f6","Type":"ContainerDied","Data":"07d7f8c4d11cff94010d44a86211ffb708e184b274a7cd9c01fac838c2c30abf"}
Nov 03 20:38:09.424238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.420222    5282 generic.go:356] "Generic (PLEG): container finished" podID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerID="caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a" exitCode=0
Nov 03 20:38:09.424238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.420271    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" event={"ID":"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4","Type":"ContainerDied","Data":"caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a"}
Nov 03 20:38:09.424238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.420288    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"
Nov 03 20:38:09.424238 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.420310    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7" event={"ID":"59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4","Type":"ContainerDied","Data":"c72bd7329508f657651888aa76508a2a2f9b33736c95498ae401145052bc2e92"}
Nov 03 20:38:09.429318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.429300    5282 scope.go:122] "RemoveContainer" containerID="8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee"
Nov 03 20:38:09.429628 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:38:09.429608    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee\": container with ID starting with 8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee not found: ID does not exist" containerID="8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee"
Nov 03 20:38:09.429698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.429635    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee"} err="failed to get container status \"8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee\": rpc error: code = NotFound desc = could not find container \"8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee\": container with ID starting with 8df53d829a94edad21e5b76ddd4223b5f9b981936461fbfcef7196bf9a6872ee not found: ID does not exist"
Nov 03 20:38:09.429698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.429649    5282 scope.go:122] "RemoveContainer" containerID="c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823"
Nov 03 20:38:09.458840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.458816    5282 scope.go:122] "RemoveContainer" containerID="c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823"
Nov 03 20:38:09.459118 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:38:09.459098    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823\": container with ID starting with c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823 not found: ID does not exist" containerID="c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823"
Nov 03 20:38:09.459192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.459127    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823"} err="failed to get container status \"c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823\": rpc error: code = NotFound desc = could not find container \"c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823\": container with ID starting with c5e00312085fe246fb27965b3a0f04ebc4edc3315aaf0bee454293d5a53be823 not found: ID does not exist"
Nov 03 20:38:09.459192 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.459145    5282 scope.go:122] "RemoveContainer" containerID="93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a"
Nov 03 20:38:09.466222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.466169    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"]
Nov 03 20:38:09.467677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.467661    5282 scope.go:122] "RemoveContainer" containerID="93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a"
Nov 03 20:38:09.467943 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:38:09.467921    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a\": container with ID starting with 93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a not found: ID does not exist" containerID="93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a"
Nov 03 20:38:09.468011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.467951    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a"} err="failed to get container status \"93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a\": rpc error: code = NotFound desc = could not find container \"93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a\": container with ID starting with 93dca24e43f779f6739a565b681d200ad84a78af3f0e03a981d140694970853a not found: ID does not exist"
Nov 03 20:38:09.468011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.467970    5282 scope.go:122] "RemoveContainer" containerID="caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a"
Nov 03 20:38:09.470042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.470020    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-45b2b"]
Nov 03 20:38:09.476036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.476018    5282 scope.go:122] "RemoveContainer" containerID="caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a"
Nov 03 20:38:09.476343 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:38:09.476316    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a\": container with ID starting with caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a not found: ID does not exist" containerID="caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a"
Nov 03 20:38:09.476405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.476350    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a"} err="failed to get container status \"caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a\": rpc error: code = NotFound desc = could not find container \"caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a\": container with ID starting with caf6c99ca5883c40b7daa9a0f1da63ae653664de046eee3921acf1c04f79f49a not found: ID does not exist"
Nov 03 20:38:09.480230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.480183    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"]
Nov 03 20:38:09.485902 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.485879    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-7krc7"]
Nov 03 20:38:09.496509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.496487    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"]
Nov 03 20:38:09.500332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.500309    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cwz84"]
Nov 03 20:38:09.509874 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.509851    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"]
Nov 03 20:38:09.513923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:09.513901    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2n8gb"]
Nov 03 20:38:10.533025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:10.532992    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:38:11.452844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:11.452810    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" path="/var/lib/kubelet/pods/35226e43-d541-43f8-a888-edb3202ad0f6/volumes"
Nov 03 20:38:11.453261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:11.453248    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" path="/var/lib/kubelet/pods/59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4/volumes"
Nov 03 20:38:11.453653 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:11.453641    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" path="/var/lib/kubelet/pods/b5cc1995-f631-409c-afd1-a3cd057a51a2/volumes"
Nov 03 20:38:11.454353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:11.454340    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" path="/var/lib/kubelet/pods/c607312e-afa0-4bcf-8885-c2862a324ed8/volumes"
Nov 03 20:38:14.373150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.373114    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:14.719465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719381    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"]
Nov 03 20:38:14.719925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719907    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy"
Nov 03 20:38:14.719981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719927    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy"
Nov 03 20:38:14.719981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719944    5282 cpu_manager.go:414] "containerMap: removing container" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy"
Nov 03 20:38:14.719981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719951    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy"
Nov 03 20:38:14.719981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719972    5282 cpu_manager.go:414] "containerMap: removing container" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy"
Nov 03 20:38:14.719981 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719977    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy"
Nov 03 20:38:14.720188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719988    5282 cpu_manager.go:414] "containerMap: removing container" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy"
Nov 03 20:38:14.720188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.719993    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy"
Nov 03 20:38:14.720188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.720056    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c607312e-afa0-4bcf-8885-c2862a324ed8" containerName="istio-proxy"
Nov 03 20:38:14.720188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.720064    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cc1995-f631-409c-afd1-a3cd057a51a2" containerName="istio-proxy"
Nov 03 20:38:14.720188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.720074    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="35226e43-d541-43f8-a888-edb3202ad0f6" containerName="istio-proxy"
Nov 03 20:38:14.720188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.720081    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="59cf0dfd-822a-451b-8c1d-e2c3ce62f7f4" containerName="istio-proxy"
Nov 03 20:38:14.724536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.724511    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.728870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.728844    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4442a4-openshift-default-dockercfg-pc727\""
Nov 03 20:38:14.737034 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.735186    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"]
Nov 03 20:38:14.791527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.791498    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:14.810723 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.810687    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-socket\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.810865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.810749    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgk8\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-kube-api-access-9rgk8\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.810865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.810831    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-data\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.810970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.810874    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-credential-socket\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.810970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.810937    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84607dcb-0cee-493e-991c-41416db3e7fb-istiod-ca-cert\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.811068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.810968    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84607dcb-0cee-493e-991c-41416db3e7fb-istio-podinfo\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.811068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.810991    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-certs\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.811068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.811016    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-envoy\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.811190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.811098    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-istio-token\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.905061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.905027    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:14.912298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912266    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-socket\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912335    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgk8\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-kube-api-access-9rgk8\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912363    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-data\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912405    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-credential-socket\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912436    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84607dcb-0cee-493e-991c-41416db3e7fb-istiod-ca-cert\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912466    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84607dcb-0cee-493e-991c-41416db3e7fb-istio-podinfo\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912563    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-certs\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912610    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-envoy\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912673    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-istio-token\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.912926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.912841    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-socket\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.913626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.913050    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-credential-socket\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.913626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.913165    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-certs\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.913626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.913337    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-data\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.913626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.913595    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84607dcb-0cee-493e-991c-41416db3e7fb-istiod-ca-cert\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.915086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.915057    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-envoy\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.915191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.915173    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84607dcb-0cee-493e-991c-41416db3e7fb-istio-podinfo\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.923031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.923006    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-istio-token\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:14.923109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:14.923051    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgk8\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-kube-api-access-9rgk8\") pod \"e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") " pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:15.041388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.041349    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:15.317553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.317435    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"]
Nov 03 20:38:15.319886 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:38:15.319858    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84607dcb_0cee_493e_991c_41416db3e7fb.slice/crio-5bd36667f4cfd6163fde9ba7ab907d4fe32e7aad0949ed3b21363fa3fdb1d93b WatchSource:0}: Error finding container 5bd36667f4cfd6163fde9ba7ab907d4fe32e7aad0949ed3b21363fa3fdb1d93b: Status 404 returned error can't find the container with id 5bd36667f4cfd6163fde9ba7ab907d4fe32e7aad0949ed3b21363fa3fdb1d93b
Nov 03 20:38:15.322125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.322088    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:15.322255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.322171    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:15.322329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.322268    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:15.453577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.453540    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" event={"ID":"84607dcb-0cee-493e-991c-41416db3e7fb","Type":"ContainerStarted","Data":"2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0"}
Nov 03 20:38:15.453938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.453591    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" event={"ID":"84607dcb-0cee-493e-991c-41416db3e7fb","Type":"ContainerStarted","Data":"5bd36667f4cfd6163fde9ba7ab907d4fe32e7aad0949ed3b21363fa3fdb1d93b"}
Nov 03 20:38:15.473416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:15.473358    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podStartSLOduration=1.473340903 podStartE2EDuration="1.473340903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:15.472625996 +0000 UTC m=+3488.141827508" watchObservedRunningTime="2026-11-03 20:38:15.473340903 +0000 UTC m=+3488.142542375"
Nov 03 20:38:16.041693 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:16.041657    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:16.046737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:16.046710    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:16.461303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:16.458870    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:16.462353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:16.462323    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:17.683483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:17.683443    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:17.817471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:17.817434    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:18.433743 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.433701    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"]
Nov 03 20:38:18.437332 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.437312    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.441190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.441141    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dkm76\""
Nov 03 20:38:18.447590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.447562    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"]
Nov 03 20:38:18.550012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.549972    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550020    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550114    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550165    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550309    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550343    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550378    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vc8g\" (UniqueName: \"kubernetes.io/projected/f0ac7456-d600-48bd-b83d-8ed330bf70c0-kube-api-access-5vc8g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550399    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.550479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.550415    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652060 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652026    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652092    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652119    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652142    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652157    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vc8g\" (UniqueName: \"kubernetes.io/projected/f0ac7456-d600-48bd-b83d-8ed330bf70c0-kube-api-access-5vc8g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652177    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652225    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652293    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652320    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652749    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652765    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652895    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.652985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.652894    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.653314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.653290    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.654418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.654398    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.654904 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.654888    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.662830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.662806    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ac7456-d600-48bd-b83d-8ed330bf70c0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.663048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.663027    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vc8g\" (UniqueName: \"kubernetes.io/projected/f0ac7456-d600-48bd-b83d-8ed330bf70c0-kube-api-access-5vc8g\") pod \"maas-default-gateway-openshift-default-fcb75c45f-krhrv\" (UID: \"f0ac7456-d600-48bd-b83d-8ed330bf70c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.753666 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.753627    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:18.888744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:18.888707    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:19.028460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.028360    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:19.036290 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.036263    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"]
Nov 03 20:38:19.038463 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:38:19.038436    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ac7456_d600_48bd_b83d_8ed330bf70c0.slice/crio-68a74c5dec8802359d81c07236dcac3d97031a2c078105dd93fe536579edfb8e WatchSource:0}: Error finding container 68a74c5dec8802359d81c07236dcac3d97031a2c078105dd93fe536579edfb8e: Status 404 returned error can't find the container with id 68a74c5dec8802359d81c07236dcac3d97031a2c078105dd93fe536579edfb8e
Nov 03 20:38:19.041707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.041649    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:19.041832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.041744    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:19.041832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.041789    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:19.148728 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.148695    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:19.481550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.481164    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" event={"ID":"f0ac7456-d600-48bd-b83d-8ed330bf70c0","Type":"ContainerStarted","Data":"bc5c80273d31a1016d381ef0185bda2c13eb992eec81f0bee6a1e7c3e146f330"}
Nov 03 20:38:19.481550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.481271    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" event={"ID":"f0ac7456-d600-48bd-b83d-8ed330bf70c0","Type":"ContainerStarted","Data":"68a74c5dec8802359d81c07236dcac3d97031a2c078105dd93fe536579edfb8e"}
Nov 03 20:38:19.507269 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.504174    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podStartSLOduration=1.5041486929999999 podStartE2EDuration="1.504148693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:19.502587558 +0000 UTC m=+3492.171789030" watchObservedRunningTime="2026-11-03 20:38:19.504148693 +0000 UTC m=+3492.173350195"
Nov 03 20:38:19.753881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.753796    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:19.860989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:19.860958    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:20.555931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:20.555902    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:20.755691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:20.755656    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 03 20:38:20.756105 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:20.755712    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 03 20:38:21.292129 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:21.292094    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:21.754472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:21.754434    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 03 20:38:21.754631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:21.754488    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 03 20:38:21.799839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:21.799804    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:38:22.164080 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:22.163999    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:22.627658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:22.627626    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:22.754748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:22.754704    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 03 20:38:22.755057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:22.755026    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 03 20:38:23.019177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:23.019144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:23.137716 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:23.137674    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:23.754557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:23.754505    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 03 20:38:23.754765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:23.754602    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 03 20:38:23.921591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:23.921556    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:24.696222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:24.696000    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:24.754689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:24.754583    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 03 20:38:24.754689 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:24.754652    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 03 20:38:25.474061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:25.474027    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:25.754164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:25.754128    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 03 20:38:25.754635 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:25.754192    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 03 20:38:26.149155 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:26.149077    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:26.754752 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:26.754718    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body=
Nov 03 20:38:26.755177 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:26.754782    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused"
Nov 03 20:38:26.864150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:26.864097    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:27.569295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:27.569243    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:27.864124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:27.864046    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:27.864612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:27.864590    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:27.865453 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:27.865437    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv"
Nov 03 20:38:28.320262 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:28.320228    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:29.025320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:29.025283    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:29.747284 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:29.747181    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:30.473081 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:30.473047    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:31.122596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.122551    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:31.460536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.460498    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 20:38:31.460762 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.460565    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 20:38:31.575033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.574935    5282 generic.go:356] "Generic (PLEG): container finished" podID="84607dcb-0cee-493e-991c-41416db3e7fb" containerID="2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0" exitCode=0
Nov 03 20:38:31.575033 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.574993    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" event={"ID":"84607dcb-0cee-493e-991c-41416db3e7fb","Type":"ContainerDied","Data":"2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0"}
Nov 03 20:38:31.575609 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.575493    5282 scope.go:122] "RemoveContainer" containerID="2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0"
Nov 03 20:38:31.576804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.576773    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:31.576879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.576845    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:31.576921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.576879    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:38:31.622290 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:38:31.622249    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:38:31.744909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:31.744865    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:32.444530 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:32.444493    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:32.600669 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:32.600617    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" event={"ID":"84607dcb-0cee-493e-991c-41416db3e7fb","Type":"ContainerStarted","Data":"dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595"}
Nov 03 20:38:32.662116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:32.662079    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:33.042329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:33.042292    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:33.042705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:33.042682    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 20:38:33.042816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:33.042737    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 20:38:33.267764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:33.267716    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:33.400558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:33.400453    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:33.554463 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:33.554429    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:34.042606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:34.042557    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 20:38:34.043038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:34.042632    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 20:38:34.240085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:34.240042    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:34.661433 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:34.661403    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:35.030846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:35.030811    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:35.041982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:35.041954    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 20:38:35.042109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:35.042001    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 20:38:35.846159 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:35.846123    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:36.042769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:36.042731    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 20:38:36.042964 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:36.042789    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 20:38:36.191700 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:36.191600    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:38:36.765149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:36.765115    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:37.042456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:37.042364    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 20:38:37.042456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:37.042424    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 20:38:37.549185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:37.549139    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:38.075961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:38.075920    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:38.076522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:38.076466    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:38.077462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:38.077435    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:38:38.308868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:38.308814    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:39.015652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:39.015614    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:39.628994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:39.628949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:40.224153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:40.224117    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:40.836373 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:40.836338    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:41.478862 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:41.478827    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:42.124945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:42.124909    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:42.755612 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:42.755556    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:43.403467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:43.403435    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:44.031445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:44.031413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:44.627828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:44.627798    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:45.379654 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:45.379624    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:46.108075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:46.108043    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:46.834070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:46.834038    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:47.538761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:47.538723    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:48.154054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:48.154018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:48.794181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:48.794145    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:49.433946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:49.433915    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:50.029656 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:50.029618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:50.648770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:50.648738    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:51.315031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:51.314999    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:51.954071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:51.954034    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:52.729627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:52.729594    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:52.889775 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:52.889736    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:38:53.421985 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:53.421943    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:54.042451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:54.042404    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:54.727839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:54.727806    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:55.472166 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:55.472129    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:56.258050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:56.258019    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:56.850241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:56.850179    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:57.628445 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:57.628397    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:58.243719 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:58.243670    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:59.024948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:59.024916    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:38:59.698452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:38:59.698415    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:00.317089 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:00.317058    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:00.924442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:00.924376    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:01.570871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:01.570832    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:01.614064 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:39:01.614029    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:39:02.271847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:02.271816    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:02.929645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:02.929615    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:03.612736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:03.612704    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:04.235184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:04.235147    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:04.926456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:04.926426    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:05.566793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:05.566761    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:06.144474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:06.144439    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:06.902164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:06.902134    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:07.525926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:07.525887    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:08.129744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:08.129709    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:08.742541 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:08.742506    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:09.339477 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:09.339440    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:09.935625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:09.935588    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:10.635164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:10.635131    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:11.237496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:11.237461    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:11.830976 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:11.830944    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:12.432118 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:12.432088    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:13.083237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:13.083185    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:13.734945 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:13.734909    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:14.416869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:14.416835    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:15.108146 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:15.108115    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:15.458575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:15.458495    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:15.993312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:15.993279    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:16.453138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:16.453044    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:39:16.736679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:16.736649    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:17.466676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:17.466641    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:18.213224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:18.213172    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:18.832882 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:18.832847    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:19.509886 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:19.509852    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:20.224552 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:20.224519    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:20.484841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:20.484813    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:20.835590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:20.835510    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:21.626370 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:21.626337    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:22.315830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:22.315801    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:22.684917 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:22.684837    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:22.928678 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:22.928643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:23.633135 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:23.633104    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:24.335568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:24.335539    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:24.925024 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:24.924994    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:25.753936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:25.753894    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:25.879316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:25.879286    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:26.420522 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:26.420488    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:26.925375 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:26.925340    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:27.061563 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:27.061527    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:27.183300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:27.183216    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:27.606285 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:27.606250    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:28.439804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:28.439761    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:39:31.612527 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:39:31.612491    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:39:33.320549 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:33.320514    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:33.624246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:33.624127    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"]
Nov 03 20:39:33.624437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:33.624403    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" containerID="cri-o://dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595" gracePeriod=30
Nov 03 20:39:33.628715 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:33.628682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:33.747637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:33.747603    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:38.078936 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.078898    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 20:39:38.079345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.078958    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:39:38.868692 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.868663    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:39:38.899614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.899534    5282 generic.go:356] "Generic (PLEG): container finished" podID="84607dcb-0cee-493e-991c-41416db3e7fb" containerID="dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595" exitCode=0
Nov 03 20:39:38.899614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.899583    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" event={"ID":"84607dcb-0cee-493e-991c-41416db3e7fb","Type":"ContainerDied","Data":"dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595"}
Nov 03 20:39:38.899849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.899651    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h" event={"ID":"84607dcb-0cee-493e-991c-41416db3e7fb","Type":"ContainerDied","Data":"5bd36667f4cfd6163fde9ba7ab907d4fe32e7aad0949ed3b21363fa3fdb1d93b"}
Nov 03 20:39:38.899849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.899585    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"
Nov 03 20:39:38.899849 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.899673    5282 scope.go:122] "RemoveContainer" containerID="dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595"
Nov 03 20:39:38.908022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.907997    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-data\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908035    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84607dcb-0cee-493e-991c-41416db3e7fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84607dcb-0cee-493e-991c-41416db3e7fb-istio-podinfo\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908090    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-certs\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908171 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908129    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-kube-api-access-9rgk8\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-kube-api-access-9rgk8\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908189    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-credential-socket\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908252    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-istio-token\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-istio-token\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908300    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-envoy\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908368 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908348    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84607dcb-0cee-493e-991c-41416db3e7fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84607dcb-0cee-493e-991c-41416db3e7fb-istiod-ca-cert\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908398    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-socket\") pod \"84607dcb-0cee-493e-991c-41416db3e7fb\" (UID: \"84607dcb-0cee-493e-991c-41416db3e7fb\") "
Nov 03 20:39:38.908571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908405    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-data" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.908639 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908598    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-credential-socket" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.908732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908665    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-certs" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.908732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908714    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.908827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908734    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.909007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908985    5282 scope.go:122] "RemoveContainer" containerID="2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0"
Nov 03 20:39:38.909246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.908987    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84607dcb-0cee-493e-991c-41416db3e7fb-istiod-ca-cert" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:38.909877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.909853    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-socket" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.911003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.910976    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-envoy" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.911087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.911018    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-kube-api-access-9rgk8" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "kube-api-access-9rgk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:38.911087 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.911033    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84607dcb-0cee-493e-991c-41416db3e7fb-istio-podinfo" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:38.911276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.911246    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-istio-token" pod "84607dcb-0cee-493e-991c-41416db3e7fb" (UID: "84607dcb-0cee-493e-991c-41416db3e7fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:38.931380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.931355    5282 scope.go:122] "RemoveContainer" containerID="dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595"
Nov 03 20:39:38.931653 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:39:38.931632    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595\": container with ID starting with dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595 not found: ID does not exist" containerID="dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595"
Nov 03 20:39:38.931706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.931666    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595"} err="failed to get container status \"dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595\": rpc error: code = NotFound desc = could not find container \"dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595\": container with ID starting with dc28b71ff34c2d758d0e0173157d16afcc786f5772dbd4722c1e4da75c098595 not found: ID does not exist"
Nov 03 20:39:38.931706 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.931689    5282 scope.go:122] "RemoveContainer" containerID="2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0"
Nov 03 20:39:38.931880 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:39:38.931864    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0\": container with ID starting with 2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0 not found: ID does not exist" containerID="2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0"
Nov 03 20:39:38.931925 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:38.931885    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0"} err="failed to get container status \"2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0\": rpc error: code = NotFound desc = could not find container \"2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0\": container with ID starting with 2c349ebc37af2216350e631fbbfda07856ffd1c3ac5f12639cb049cc9ab412f0 not found: ID does not exist"
Nov 03 20:39:39.009593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.009558    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:39.009593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.009589    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgk8\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-kube-api-access-9rgk8\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:39.009593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.009599    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84607dcb-0cee-493e-991c-41416db3e7fb-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:39.009815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.009608    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:39.009815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.009617    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84607dcb-0cee-493e-991c-41416db3e7fb-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:39.009815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.009626    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84607dcb-0cee-493e-991c-41416db3e7fb-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:39.009815 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.009634    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84607dcb-0cee-493e-991c-41416db3e7fb-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:39.221153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.221124    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"]
Nov 03 20:39:39.226927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.226895    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4442a4-openshift-default-b7d765c97-rcf2h"]
Nov 03 20:39:39.453064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:39.453028    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" path="/var/lib/kubelet/pods/84607dcb-0cee-493e-991c-41416db3e7fb/volumes"
Nov 03 20:39:40.586289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:40.586238    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:40.704032 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:40.704000    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:46.247054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:46.247019    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:39:50.288295 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:50.288259    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:39:51.862769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:51.862732    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:52.734756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:52.734720    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:39:58.489366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:39:58.489330    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:01.613841 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:40:01.613801    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:40:03.588189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.588149    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"]
Nov 03 20:40:03.588765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.588744    5282 cpu_manager.go:414] "containerMap: removing container" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy"
Nov 03 20:40:03.588844 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.588769    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy"
Nov 03 20:40:03.588911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.588886    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy"
Nov 03 20:40:03.588911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.588905    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy"
Nov 03 20:40:03.592096 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.592072    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.605231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.605172    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"]
Nov 03 20:40:03.623224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.623177    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"]
Nov 03 20:40:03.623587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.623568    5282 cpu_manager.go:414] "containerMap: removing container" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy"
Nov 03 20:40:03.623587 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.623586    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84607dcb-0cee-493e-991c-41416db3e7fb" containerName="istio-proxy"
Nov 03 20:40:03.626827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.626804    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.640287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.640258    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"]
Nov 03 20:40:03.733883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.733846    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.733890    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734055 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.733918    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734067    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734136    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpd8\" (UniqueName: \"kubernetes.io/projected/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-kube-api-access-7xpd8\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734166    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734185    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734251    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734350 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734293    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734348    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734381    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734399    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.734457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734423    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734457 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734450    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734484    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a446f2-8afb-4a18-9625-ef1c817a0883-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734539    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734567    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9hvn\" (UniqueName: \"kubernetes.io/projected/39a446f2-8afb-4a18-9625-ef1c817a0883-kube-api-access-g9hvn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.734623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.734588    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.835438 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835401    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.835615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835446    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.835615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835462    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.835615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835478    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.835615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835503    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.835615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835518    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.835615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835539    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a446f2-8afb-4a18-9625-ef1c817a0883-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.835615 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835562    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.836093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835927    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.836093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835953    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.836093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835983    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835990    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.836093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.835995    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9hvn\" (UniqueName: \"kubernetes.io/projected/39a446f2-8afb-4a18-9625-ef1c817a0883-kube-api-access-g9hvn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.836093 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836070    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836109    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836155    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836221    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836250    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836284    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpd8\" (UniqueName: \"kubernetes.io/projected/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-kube-api-access-7xpd8\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836321    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836326    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836387    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836418    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836460    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.836497 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836472    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a446f2-8afb-4a18-9625-ef1c817a0883-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.837074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836601    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.837074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836750    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.837074 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.836886    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.838478 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.838423    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.838643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.838628    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.838876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.838858    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.838934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.838916    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.844002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.843975    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a446f2-8afb-4a18-9625-ef1c817a0883-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.844137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.844117    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.844296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.844180    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9hvn\" (UniqueName: \"kubernetes.io/projected/39a446f2-8afb-4a18-9625-ef1c817a0883-kube-api-access-g9hvn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-fj62v\" (UID: \"39a446f2-8afb-4a18-9625-ef1c817a0883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.844601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.844582    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpd8\" (UniqueName: \"kubernetes.io/projected/1f0af13b-f24a-4817-a5da-a4f9ea8402ac-kube-api-access-7xpd8\") pod \"maas-default-gateway-openshift-default-fcb75c45f-2cl2z\" (UID: \"1f0af13b-f24a-4817-a5da-a4f9ea8402ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:03.905816 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.905761    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:03.942011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:03.941978    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:04.181402 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.181374    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"]
Nov 03 20:40:04.184282 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:40:04.184255    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a446f2_8afb_4a18_9625_ef1c817a0883.slice/crio-2335ccdcee9b80426a0a67e84694ed9fadff5108182e5c0ecea3e1cf8513065f WatchSource:0}: Error finding container 2335ccdcee9b80426a0a67e84694ed9fadff5108182e5c0ecea3e1cf8513065f: Status 404 returned error can't find the container with id 2335ccdcee9b80426a0a67e84694ed9fadff5108182e5c0ecea3e1cf8513065f
Nov 03 20:40:04.186349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.186315    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:40:04.186431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.186385    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:40:04.186431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.186414    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:40:04.226479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.226450    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"]
Nov 03 20:40:04.229292 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:40:04.229266    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0af13b_f24a_4817_a5da_a4f9ea8402ac.slice/crio-b84f07f810fea37914ae58b575055df0ac4244a72873e4b1f72ce0cedcdb2371 WatchSource:0}: Error finding container b84f07f810fea37914ae58b575055df0ac4244a72873e4b1f72ce0cedcdb2371: Status 404 returned error can't find the container with id b84f07f810fea37914ae58b575055df0ac4244a72873e4b1f72ce0cedcdb2371
Nov 03 20:40:04.231455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.231424    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:40:04.231571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.231501    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:40:04.231571 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:04.231548    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:40:05.027340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.027298    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" event={"ID":"1f0af13b-f24a-4817-a5da-a4f9ea8402ac","Type":"ContainerStarted","Data":"8021a70ead809961ee9467acd896514f7d7132768b93bc9e1b0b475a7a62a496"}
Nov 03 20:40:05.027834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.027353    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" event={"ID":"1f0af13b-f24a-4817-a5da-a4f9ea8402ac","Type":"ContainerStarted","Data":"b84f07f810fea37914ae58b575055df0ac4244a72873e4b1f72ce0cedcdb2371"}
Nov 03 20:40:05.028805 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.028782    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" event={"ID":"39a446f2-8afb-4a18-9625-ef1c817a0883","Type":"ContainerStarted","Data":"abf2c55703bd721053bfc4d7c22b23566d479595757ee9878474b5aee29e795b"}
Nov 03 20:40:05.028908 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.028832    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" event={"ID":"39a446f2-8afb-4a18-9625-ef1c817a0883","Type":"ContainerStarted","Data":"2335ccdcee9b80426a0a67e84694ed9fadff5108182e5c0ecea3e1cf8513065f"}
Nov 03 20:40:05.047866 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.047821    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podStartSLOduration=2.0478088740000002 podStartE2EDuration="2.047808874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:05.047518085 +0000 UTC m=+3597.716719557" watchObservedRunningTime="2026-11-03 20:40:05.047808874 +0000 UTC m=+3597.717010341"
Nov 03 20:40:05.068313 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.068274    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podStartSLOduration=2.068257364 podStartE2EDuration="2.068257364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:05.067470568 +0000 UTC m=+3597.736672039" watchObservedRunningTime="2026-11-03 20:40:05.068257364 +0000 UTC m=+3597.737458834"
Nov 03 20:40:05.906228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.906179    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:05.911254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.911223    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:05.942622 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.942591    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:05.947504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:05.947479    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:06.033644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:06.033602    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:06.034075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:06.033662    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:06.034537 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:06.034513    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:40:06.034657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:06.034570    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:40:22.734868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:22.734777    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:31.621086 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:40:31.621038    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:40:34.170543 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:34.170503    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:40:36.031245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:40:36.031188    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:01.612979 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:41:01.612941    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:41:09.942025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:09.941990    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:22.785466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:22.785431    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:41:29.955242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:29.955192    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:41:31.659307 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:41:31.659261    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:41:40.483076 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.483040    5282 generic.go:356] "Generic (PLEG): container finished" podID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerID="8021a70ead809961ee9467acd896514f7d7132768b93bc9e1b0b475a7a62a496" exitCode=0
Nov 03 20:41:40.483575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.483112    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" event={"ID":"1f0af13b-f24a-4817-a5da-a4f9ea8402ac","Type":"ContainerDied","Data":"8021a70ead809961ee9467acd896514f7d7132768b93bc9e1b0b475a7a62a496"}
Nov 03 20:41:40.483575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.483523    5282 scope.go:122] "RemoveContainer" containerID="8021a70ead809961ee9467acd896514f7d7132768b93bc9e1b0b475a7a62a496"
Nov 03 20:41:40.484714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.484677    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:41:40.484867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.484772    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:41:40.484867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.484779    5282 generic.go:356] "Generic (PLEG): container finished" podID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerID="abf2c55703bd721053bfc4d7c22b23566d479595757ee9878474b5aee29e795b" exitCode=0
Nov 03 20:41:40.484867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.484822    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:41:40.484867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.484829    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" event={"ID":"39a446f2-8afb-4a18-9625-ef1c817a0883","Type":"ContainerDied","Data":"abf2c55703bd721053bfc4d7c22b23566d479595757ee9878474b5aee29e795b"}
Nov 03 20:41:40.485187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.485170    5282 scope.go:122] "RemoveContainer" containerID="abf2c55703bd721053bfc4d7c22b23566d479595757ee9878474b5aee29e795b"
Nov 03 20:41:40.486139 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.486112    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:41:40.486263 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.486185    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:41:40.486331 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:40.486257    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:41:41.524067 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.523999    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" event={"ID":"1f0af13b-f24a-4817-a5da-a4f9ea8402ac","Type":"ContainerStarted","Data":"17540759489a21d82f6147ade3820e08774ff5425415bf66662c81303d05745a"}
Nov 03 20:41:41.526030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.526004    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" event={"ID":"39a446f2-8afb-4a18-9625-ef1c817a0883","Type":"ContainerStarted","Data":"37c4178810ab829513c0a5d6717f86e1bb5f131ae0ffba89ccb937631aeb262e"}
Nov 03 20:41:41.909738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.909649    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:41:41.910104 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.910008    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:41:41.910288 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.910266    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 20:41:41.942732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.942681    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:41:41.943007 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.942981    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:41:41.943090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:41.943025    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 20:41:42.909222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:42.908608    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:41:42.909222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:42.908711    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 20:41:42.942536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:42.942502    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:41:42.942724 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:42.942550    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 20:41:43.907643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:43.907600    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:41:43.907938 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:43.907907    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 20:41:43.943568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:43.943537    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:41:43.943986 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:43.943598    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 20:41:44.913705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:44.913657    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:41:44.913927 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:44.913729    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 20:41:44.943061 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:44.943026    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:41:44.943261 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:44.943081    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 20:41:45.906679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:45.906633    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:41:45.907415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:45.907376    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 20:41:45.942660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:45.942627    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:41:45.942833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:45.942679    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 20:41:46.906719 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:46.906674    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:41:46.907124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:46.906745    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 20:41:46.943403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:46.943363    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:41:46.943568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:46.943428    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 20:41:47.906233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:47.906171    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:41:47.906416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:47.906271    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 03 20:41:47.942995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:47.942957    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:41:47.943372 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:47.943015    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Nov 03 20:41:48.910858 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:48.910829    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:41:48.911111 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:48.911080    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:41:48.911704 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:48.911685    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v"
Nov 03 20:41:48.946398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:48.946368    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:41:48.947121 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:48.947098    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:41:48.947544 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:48.947521    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z"
Nov 03 20:41:54.750051 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:54.750014    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:41:59.771643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:41:59.771605    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:42:01.613828 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:42:01.613790    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:42:13.583798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:13.583754    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:42:14.378989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.378945    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.697783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.697691    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"]
Nov 03 20:42:14.701727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.701702    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.705191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.705166    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4b0085-openshift-default-dockercfg-mvw6n\""
Nov 03 20:42:14.712001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.711977    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"]
Nov 03 20:42:14.773717 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.773666    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.863056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863016    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmqqt\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-kube-api-access-qmqqt\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863407    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-credential-socket\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863461    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-certs\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863552    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-envoy\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863579    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-token\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863612    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-podinfo\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863639    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-socket\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863704    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e5115a-f841-460f-83b5-7800a6ac4f01-istiod-ca-cert\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.863906 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.863863    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-data\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.893190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.893155    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:14.964688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964603    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e5115a-f841-460f-83b5-7800a6ac4f01-istiod-ca-cert\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964648    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-data\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964682    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqqt\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-kube-api-access-qmqqt\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964726    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-credential-socket\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964755    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-certs\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964831    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-envoy\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964854    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-token\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964883    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-podinfo\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.964993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.964912    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-socket\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.965314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.965120    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-data\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.965314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.965174    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-certs\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.965314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.965247    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-credential-socket\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.965482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.965399    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-socket\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.965482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.965432    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e5115a-f841-460f-83b5-7800a6ac4f01-istiod-ca-cert\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.967209 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.967171    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-envoy\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.967412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.967394    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-podinfo\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.972250 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.972225    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-token\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:14.972521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:14.972504    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqqt\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-kube-api-access-qmqqt\") pod \"e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") " pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:15.018933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.018907    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:15.297450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.297423    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"]
Nov 03 20:42:15.299949 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:42:15.299921    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e5115a_f841_460f_83b5_7800a6ac4f01.slice/crio-eba887648f88edb1704a9c71f4fb89385c33c4801f57aa33317b5d96d26ceeb9 WatchSource:0}: Error finding container eba887648f88edb1704a9c71f4fb89385c33c4801f57aa33317b5d96d26ceeb9: Status 404 returned error can't find the container with id eba887648f88edb1704a9c71f4fb89385c33c4801f57aa33317b5d96d26ceeb9
Nov 03 20:42:15.302466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.302427    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:42:15.302534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.302515    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:42:15.302572 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.302558    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:42:15.704327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.704241    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" event={"ID":"49e5115a-f841-460f-83b5-7800a6ac4f01","Type":"ContainerStarted","Data":"68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07"}
Nov 03 20:42:15.704327 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.704297    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" event={"ID":"49e5115a-f841-460f-83b5-7800a6ac4f01","Type":"ContainerStarted","Data":"eba887648f88edb1704a9c71f4fb89385c33c4801f57aa33317b5d96d26ceeb9"}
Nov 03 20:42:15.724722 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:15.724662    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podStartSLOduration=1.724648706 podStartE2EDuration="1.724648706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:15.723440006 +0000 UTC m=+3728.392641477" watchObservedRunningTime="2026-11-03 20:42:15.724648706 +0000 UTC m=+3728.393850167"
Nov 03 20:42:16.019610 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:16.019573    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:16.024912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:16.024883    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:16.708345 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:16.708316    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:16.709247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:16.709224    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:17.968450 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:17.968414    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:18.233880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:18.233850    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:19.140482 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.140448    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:19.251173 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.251136    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"]
Nov 03 20:42:19.254779 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.254760    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.269002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.268976    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-46a4c734-dockercfg-qbknb\""
Nov 03 20:42:19.270128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.270107    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"]
Nov 03 20:42:19.291968 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.291939    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:19.323223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.313311    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdmp\" (UniqueName: \"kubernetes.io/projected/e730a577-ee4e-4ebc-8593-ec3b1b065371-kube-api-access-hxdmp\") pod \"payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.323223 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.313441    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e730a577-ee4e-4ebc-8593-ec3b1b065371-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.323786 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.323755    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4b0085\""
Nov 03 20:42:19.414808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.414715    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e730a577-ee4e-4ebc-8593-ec3b1b065371-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.414808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.414784    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdmp\" (UniqueName: \"kubernetes.io/projected/e730a577-ee4e-4ebc-8593-ec3b1b065371-kube-api-access-hxdmp\") pod \"payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.415570 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.415542    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e730a577-ee4e-4ebc-8593-ec3b1b065371-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.425542 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.425520    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdmp\" (UniqueName: \"kubernetes.io/projected/e730a577-ee4e-4ebc-8593-ec3b1b065371-kube-api-access-hxdmp\") pod \"payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.567869 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.567832    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:19.843147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:19.843105    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"]
Nov 03 20:42:19.846144 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:42:19.846097    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode730a577_ee4e_4ebc_8593_ec3b1b065371.slice/crio-638237bc7d2c9933917518b8b0d5dc7a5264a21348d2b080e0c1ef8479f247d6 WatchSource:0}: Error finding container 638237bc7d2c9933917518b8b0d5dc7a5264a21348d2b080e0c1ef8479f247d6: Status 404 returned error can't find the container with id 638237bc7d2c9933917518b8b0d5dc7a5264a21348d2b080e0c1ef8479f247d6
Nov 03 20:42:20.042652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:20.042622    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:20.730411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:20.730374    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm" event={"ID":"e730a577-ee4e-4ebc-8593-ec3b1b065371","Type":"ContainerStarted","Data":"5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78"}
Nov 03 20:42:20.730855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:20.730426    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm" event={"ID":"e730a577-ee4e-4ebc-8593-ec3b1b065371","Type":"ContainerStarted","Data":"638237bc7d2c9933917518b8b0d5dc7a5264a21348d2b080e0c1ef8479f247d6"}
Nov 03 20:42:20.730855 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:20.730529    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:20.747527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:20.747480    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm" podStartSLOduration=1.747468205 podStartE2EDuration="1.747468205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:20.746450819 +0000 UTC m=+3733.415652344" watchObservedRunningTime="2026-11-03 20:42:20.747468205 +0000 UTC m=+3733.416669674"
Nov 03 20:42:20.836403 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:20.836375    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:20.976042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:20.975991    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:42:21.551125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:21.551095    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.232273 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:22.232232    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.828713 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:22.828678    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:22.953305 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:22.953259    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:23.631911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:23.631878    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:24.342072 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:24.342040    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:25.074303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:25.074264    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:25.749865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:25.749831    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:26.543517 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:26.543481    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:27.245164 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:27.245134    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:27.916342 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:27.916304    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:28.713708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:28.713670    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:29.423890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:29.423854    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:30.033527 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.033480    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:30.785419 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.785381    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:30.786876 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.786854    5282 generic.go:356] "Generic (PLEG): container finished" podID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerID="68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07" exitCode=0
Nov 03 20:42:30.786996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.786920    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" event={"ID":"49e5115a-f841-460f-83b5-7800a6ac4f01","Type":"ContainerDied","Data":"68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07"}
Nov 03 20:42:30.787322 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.787306    5282 scope.go:122] "RemoveContainer" containerID="68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07"
Nov 03 20:42:30.788623 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.788589    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:42:30.788696 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.788664    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:42:30.788738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:30.788700    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:42:31.454312 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:31.454279    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:31.621552 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:42:31.621513    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:42:31.709073 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:31.708992    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:31.737382 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:31.737354    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:42:31.805044 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:31.804997    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" event={"ID":"49e5115a-f841-460f-83b5-7800a6ac4f01","Type":"ContainerStarted","Data":"b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3"}
Nov 03 20:42:32.019268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:32.019235    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:32.020471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:32.020437    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:42:32.020592 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:32.020484    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:42:32.162038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:32.161402    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:32.934374 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:32.934336    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:33.019943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:33.019907    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:42:33.020141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:33.019982    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:42:33.062889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:33.062846    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:33.498248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:33.498019    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:33.703233 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:33.702663    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:33.922301 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:33.922222    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:34.020132 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:34.020097    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:42:34.020554 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:34.020157    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:42:34.636455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:34.636423    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:35.019998 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:35.019964    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:42:35.020180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:35.020015    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:42:35.456366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:35.456294    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:35.899830 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:35.899781    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:36.020392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:36.020345    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:42:36.020907 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:36.020413    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 20:42:36.197726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:36.197630    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:36.951216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:36.951142    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:37.024242 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:37.024189    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:37.024625 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:37.024499    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:37.025298 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:37.025273    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:42:37.380479 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:37.380441    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:42:37.551353 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:37.551317    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:38.334398 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:38.334355    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:39.086770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:39.086724    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:39.858949 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:39.858915    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:40.532365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:40.532330    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:41.224909 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:41.224876    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:41.937180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:41.937147    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:42.544755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:42.544724    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:43.216548 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:43.216514    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:43.845191 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:43.845157    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:44.611984 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:44.611950    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:45.310154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:45.310120    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:45.919329 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:45.919293    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:46.529337 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:46.529306    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:47.241292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:47.241186    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:47.814050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:47.814015    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:48.421835 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:48.421801    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:49.074614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:49.074565    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:49.751377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:49.748645    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:50.419947 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:50.419916    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:51.156365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:51.156296    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:51.819648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:51.819619    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:52.333120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:52.333080    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:42:52.746302 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:52.746257    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:53.422851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:53.422818    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:54.026593 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:54.026538    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:54.646330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:54.646283    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:55.241117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:55.241068    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:55.827143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:55.827107    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:56.444106 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:56.444044    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:57.129894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:57.129847    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:57.720630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:57.720593    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:58.360714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:58.359277    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:59.023495 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:59.023459    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:42:59.634794 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:42:59.634763    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:00.426003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:00.425959    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:01.129667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:01.129638    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:01.613219 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:43:01.613165    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:43:01.841980 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:01.841945    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:02.548186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:02.548154    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:03.228046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:03.228011    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:03.850386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:03.850358    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:04.446070 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:04.446010    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:05.230241 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:05.230209    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:05.860929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:05.860894    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:06.613744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:06.613713    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:07.229898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:07.229862    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:07.827989 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:07.827952    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:08.436184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:08.436149    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:09.046787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:09.046755    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:09.718601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:09.718567    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:10.389472 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:10.389437    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:11.047533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:11.047496    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:11.721751 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:11.721717    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:12.526997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:12.526964    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:13.127387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:13.127351    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:13.724456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:13.724420    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:14.331276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:14.331239    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:15.033948 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:15.033915    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:15.526807 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:15.526778    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:15.781851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:15.781749    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:43:15.904046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:15.904011    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:16.636082 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:16.636050    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:17.416128 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:17.416096    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:18.014923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:18.014881    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:18.632848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:18.632812    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:19.224631 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:19.224554    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:19.889381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:19.889352    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:20.542319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:20.542285    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:20.652212 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:20.652170    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:21.333454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:21.333422    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:22.079046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:22.079005    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:22.751670 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:22.751636    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:22.890110 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:22.890041    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:23.455138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:23.455099    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:24.108050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:24.108014    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:24.728516 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:24.728478    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:25.335337 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:25.335306    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:25.795028 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:25.794989    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"]
Nov 03 20:43:25.895850 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:25.895427    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:26.014745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:26.014705    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:26.159258 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:26.159164    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:26.584411 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:26.584378    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:26.722660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:26.722624    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:26.847817 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:26.847726    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:27.357580 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:27.357531    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:28.340499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:28.340455    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:43:30.795946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:30.795876    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm" podUID="e730a577-ee4e-4ebc-8593-ec3b1b065371" containerName="payload-processing" containerID="cri-o://5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78" gracePeriod=25
Nov 03 20:43:31.051399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.051333    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:43:31.094712 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.094682    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e730a577-ee4e-4ebc-8593-ec3b1b065371-kube-api-access-hxdmp\" (UniqueName: \"kubernetes.io/projected/e730a577-ee4e-4ebc-8593-ec3b1b065371-kube-api-access-hxdmp\") pod \"e730a577-ee4e-4ebc-8593-ec3b1b065371\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") "
Nov 03 20:43:31.094892 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.094797    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e730a577-ee4e-4ebc-8593-ec3b1b065371-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e730a577-ee4e-4ebc-8593-ec3b1b065371-plugins-config-volume\") pod \"e730a577-ee4e-4ebc-8593-ec3b1b065371\" (UID: \"e730a577-ee4e-4ebc-8593-ec3b1b065371\") "
Nov 03 20:43:31.095291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.095264    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e730a577-ee4e-4ebc-8593-ec3b1b065371-plugins-config-volume" pod "e730a577-ee4e-4ebc-8593-ec3b1b065371" (UID: "e730a577-ee4e-4ebc-8593-ec3b1b065371"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:43:31.096820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.096792    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e730a577-ee4e-4ebc-8593-ec3b1b065371-kube-api-access-hxdmp" pod "e730a577-ee4e-4ebc-8593-ec3b1b065371" (UID: "e730a577-ee4e-4ebc-8593-ec3b1b065371"). InnerVolumeSpecName "kube-api-access-hxdmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:31.122204 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.122171    5282 generic.go:356] "Generic (PLEG): container finished" podID="e730a577-ee4e-4ebc-8593-ec3b1b065371" containerID="5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78" exitCode=0
Nov 03 20:43:31.122366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.122257    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"
Nov 03 20:43:31.122366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.122258    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm" event={"ID":"e730a577-ee4e-4ebc-8593-ec3b1b065371","Type":"ContainerDied","Data":"5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78"}
Nov 03 20:43:31.122448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.122376    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm" event={"ID":"e730a577-ee4e-4ebc-8593-ec3b1b065371","Type":"ContainerDied","Data":"638237bc7d2c9933917518b8b0d5dc7a5264a21348d2b080e0c1ef8479f247d6"}
Nov 03 20:43:31.122448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.122417    5282 scope.go:122] "RemoveContainer" containerID="5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78"
Nov 03 20:43:31.135770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.135749    5282 scope.go:122] "RemoveContainer" containerID="5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78"
Nov 03 20:43:31.136029 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:43:31.136011    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78\": container with ID starting with 5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78 not found: ID does not exist" containerID="5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78"
Nov 03 20:43:31.136097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.136037    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78"} err="failed to get container status \"5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78\": rpc error: code = NotFound desc = could not find container \"5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78\": container with ID starting with 5531e2952fab8706c00e0de11c987b729aae410d374aad349804340dff67db78 not found: ID does not exist"
Nov 03 20:43:31.144153 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.144128    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"]
Nov 03 20:43:31.146880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.146857    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b0085-76c67d877g58vm"]
Nov 03 20:43:31.196124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.196090    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdmp\" (UniqueName: \"kubernetes.io/projected/e730a577-ee4e-4ebc-8593-ec3b1b065371-kube-api-access-hxdmp\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:31.196124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.196117    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e730a577-ee4e-4ebc-8593-ec3b1b065371-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:31.453658 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:31.453583    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e730a577-ee4e-4ebc-8593-ec3b1b065371" path="/var/lib/kubelet/pods/e730a577-ee4e-4ebc-8593-ec3b1b065371/volumes"
Nov 03 20:43:31.612906 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:43:31.612871    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:43:33.509682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:33.509638    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:33.631407 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:33.631368    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:33.804379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:33.804264    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"]
Nov 03 20:43:33.804628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:33.804584    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy" containerID="cri-o://b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3" gracePeriod=30
Nov 03 20:43:33.866822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:33.866787    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:37.027040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:37.027001    5282 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:43:37.027462 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:37.027059    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:43:38.054897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:38.054860    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:39.053919 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.053896    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:43:39.164086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.163990    5282 generic.go:356] "Generic (PLEG): container finished" podID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerID="b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3" exitCode=0
Nov 03 20:43:39.164086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.164048    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"
Nov 03 20:43:39.164086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.164075    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" event={"ID":"49e5115a-f841-460f-83b5-7800a6ac4f01","Type":"ContainerDied","Data":"b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3"}
Nov 03 20:43:39.164651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.164140    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f" event={"ID":"49e5115a-f841-460f-83b5-7800a6ac4f01","Type":"ContainerDied","Data":"eba887648f88edb1704a9c71f4fb89385c33c4801f57aa33317b5d96d26ceeb9"}
Nov 03 20:43:39.164651 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.164176    5282 scope.go:122] "RemoveContainer" containerID="b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3"
Nov 03 20:43:39.172755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.172739    5282 scope.go:122] "RemoveContainer" containerID="68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07"
Nov 03 20:43:39.180452 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.180434    5282 scope.go:122] "RemoveContainer" containerID="b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3"
Nov 03 20:43:39.180712 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:43:39.180696    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3\": container with ID starting with b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3 not found: ID does not exist" containerID="b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3"
Nov 03 20:43:39.180759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.180725    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3"} err="failed to get container status \"b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3\": rpc error: code = NotFound desc = could not find container \"b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3\": container with ID starting with b8021f4518a06865ce2d14edc65e192167b6003347c03112dac95fa11de5e1a3 not found: ID does not exist"
Nov 03 20:43:39.180759 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.180742    5282 scope.go:122] "RemoveContainer" containerID="68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07"
Nov 03 20:43:39.180992 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:43:39.180972    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07\": container with ID starting with 68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07 not found: ID does not exist" containerID="68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07"
Nov 03 20:43:39.181035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.180998    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07"} err="failed to get container status \"68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07\": rpc error: code = NotFound desc = could not find container \"68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07\": container with ID starting with 68837c6952f7bf4a9477ad615419d9edc22ce9d80477bb6e9491fdb2f5b56e07 not found: ID does not exist"
Nov 03 20:43:39.189309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189290    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e5115a-f841-460f-83b5-7800a6ac4f01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e5115a-f841-460f-83b5-7800a6ac4f01-istiod-ca-cert\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189321    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-kube-api-access-qmqqt\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-kube-api-access-qmqqt\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189353    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-socket\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189387 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189372    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-data\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189416    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-certs\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189479    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-credential-socket\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189498    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-envoy\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189533 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189523    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-token\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-token\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189757 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189561    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-podinfo\") pod \"49e5115a-f841-460f-83b5-7800a6ac4f01\" (UID: \"49e5115a-f841-460f-83b5-7800a6ac4f01\") "
Nov 03 20:43:39.189817 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189747    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-socket" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:39.189870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189803    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-credential-socket" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:39.189870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189808    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e5115a-f841-460f-83b5-7800a6ac4f01-istiod-ca-cert" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:43:39.189990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189943    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-certs" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:39.189990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.189761    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-data" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:39.190109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.190006    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.190109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.190028    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49e5115a-f841-460f-83b5-7800a6ac4f01-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.190109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.190042    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.191502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.191480    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-kube-api-access-qmqqt" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "kube-api-access-qmqqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:39.191617 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.191567    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-envoy" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:43:39.191764 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.191742    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-podinfo" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:43:39.191811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.191791    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-token" pod "49e5115a-f841-460f-83b5-7800a6ac4f01" (UID: "49e5115a-f841-460f-83b5-7800a6ac4f01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:43:39.291057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.291014    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.291057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.291049    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmqqt\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-kube-api-access-qmqqt\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.291057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.291062    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.291349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.291070    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.291349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.291080    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.291349 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.291087    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49e5115a-f841-460f-83b5-7800a6ac4f01-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:43:39.473154 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.473126    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:43:39.483038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.483010    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"]
Nov 03 20:43:39.484995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:39.484970    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b0085-openshift-default-5499b86fccm6r7f"]
Nov 03 20:43:41.456091 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:41.456057    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" path="/var/lib/kubelet/pods/49e5115a-f841-460f-83b5-7800a6ac4f01/volumes"
Nov 03 20:43:53.847014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.846978    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"]
Nov 03 20:43:53.847550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.847410    5282 cpu_manager.go:414] "containerMap: removing container" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy"
Nov 03 20:43:53.847550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.847424    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy"
Nov 03 20:43:53.847550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.847436    5282 cpu_manager.go:414] "containerMap: removing container" podUID="e730a577-ee4e-4ebc-8593-ec3b1b065371" containerName="payload-processing"
Nov 03 20:43:53.847550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.847441    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e730a577-ee4e-4ebc-8593-ec3b1b065371" containerName="payload-processing"
Nov 03 20:43:53.847550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.847522    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy"
Nov 03 20:43:53.847550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.847530    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e730a577-ee4e-4ebc-8593-ec3b1b065371" containerName="payload-processing"
Nov 03 20:43:53.847550 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.847540    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy"
Nov 03 20:43:53.850848 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.850831    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:53.855788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.855750    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c8e4e36f-kserve-self-signed-certs\""
Nov 03 20:43:53.862253 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.862227    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"]
Nov 03 20:43:53.952057 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.952014    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:53.955149 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.955129    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tls-certs\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:53.955239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.955185    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tmp-dir\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:53.955319 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.955298    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-home\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:53.955362 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.955345    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-dshm\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:53.955466 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.955452    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-model-cache\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:53.955514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:53.955499    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb85n\" (UniqueName: \"kubernetes.io/projected/8581921b-1c0a-4874-8bcf-b0fac6876a4c-kube-api-access-rb85n\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.056608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.056575    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-model-cache\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.056804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.056627    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb85n\" (UniqueName: \"kubernetes.io/projected/8581921b-1c0a-4874-8bcf-b0fac6876a4c-kube-api-access-rb85n\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.056804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.056651    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tls-certs\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.056804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.056695    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tmp-dir\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.056804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.056715    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-home\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.057039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.056868    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-dshm\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.057113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.057087    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tmp-dir\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.057186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.057114    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-home\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.057289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.057191    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-model-cache\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.059097 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.059069    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-dshm\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.059365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.059349    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tls-certs\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.064836 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.064811    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb85n\" (UniqueName: \"kubernetes.io/projected/8581921b-1c0a-4874-8bcf-b0fac6876a4c-kube-api-access-rb85n\") pod \"e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") " pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.166983 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.166894    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:54.510417 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:54.510357    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"]
Nov 03 20:43:54.512832 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:43:54.512805    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8581921b_1c0a_4874_8bcf_b0fac6876a4c.slice/crio-e7d96543b6cb0e04680f89d8c8bac0ce6f05f079ee0f3f7e35eaa47a67744be3 WatchSource:0}: Error finding container e7d96543b6cb0e04680f89d8c8bac0ce6f05f079ee0f3f7e35eaa47a67744be3: Status 404 returned error can't find the container with id e7d96543b6cb0e04680f89d8c8bac0ce6f05f079ee0f3f7e35eaa47a67744be3
Nov 03 20:43:55.278251 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:55.278215    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq" event={"ID":"8581921b-1c0a-4874-8bcf-b0fac6876a4c","Type":"ContainerStarted","Data":"f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238"}
Nov 03 20:43:55.278672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:55.278264    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq" event={"ID":"8581921b-1c0a-4874-8bcf-b0fac6876a4c","Type":"ContainerStarted","Data":"e7d96543b6cb0e04680f89d8c8bac0ce6f05f079ee0f3f7e35eaa47a67744be3"}
Nov 03 20:43:55.296228 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:55.296167    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq" podStartSLOduration=2.296154081 podStartE2EDuration="2.296154081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:55.295496791 +0000 UTC m=+3827.964698337" watchObservedRunningTime="2026-11-03 20:43:55.296154081 +0000 UTC m=+3827.965355551"
Nov 03 20:43:55.664626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:55.664543    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:43:56.167935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:56.167896    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:43:57.825281 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:57.825144    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:58.282035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:58.281934    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:43:58.909391 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:58.909337    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:43:58.910075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:43:58.910018    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:44:01.613461 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:44:01.613415    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:44:01.675823 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:01.675789    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:04.048880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:04.048840    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:04.167488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:04.167439    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:44:04.180053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:04.180024    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:44:04.352272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:04.352174    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:44:05.410672 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:05.410640    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:11.987493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:11.987457    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:44:22.390041 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:22.390010    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:22.935636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:22.935589    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:23.642871 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:23.642832    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:44:23.948123 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:23.948023    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"]
Nov 03 20:44:24.021343 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:24.021312    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:44:29.570026 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:29.566499    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:44:31.613481 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:44:31.613441    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:44:38.467878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:38.467840    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:44:38.971320 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:38.971280    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq" podUID="8581921b-1c0a-4874-8bcf-b0fac6876a4c" containerName="main" containerID="cri-o://f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238" gracePeriod=45
Nov 03 20:44:39.472206 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.472173    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:44:39.643124 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.643013    5282 generic.go:356] "Generic (PLEG): container finished" podID="8581921b-1c0a-4874-8bcf-b0fac6876a4c" containerID="f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238" exitCode=0
Nov 03 20:44:39.643347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.643180    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq" event={"ID":"8581921b-1c0a-4874-8bcf-b0fac6876a4c","Type":"ContainerDied","Data":"f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238"}
Nov 03 20:44:39.643347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.643245    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq" event={"ID":"8581921b-1c0a-4874-8bcf-b0fac6876a4c","Type":"ContainerDied","Data":"e7d96543b6cb0e04680f89d8c8bac0ce6f05f079ee0f3f7e35eaa47a67744be3"}
Nov 03 20:44:39.643347 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.643271    5282 scope.go:122] "RemoveContainer" containerID="f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238"
Nov 03 20:44:39.643553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.643439    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"
Nov 03 20:44:39.652513 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.652473    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-dshm\") pod \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") "
Nov 03 20:44:39.652677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.652571    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tls-certs\") pod \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") "
Nov 03 20:44:39.652677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.652617    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tmp-dir\") pod \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") "
Nov 03 20:44:39.652677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.652668    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8581921b-1c0a-4874-8bcf-b0fac6876a4c-kube-api-access-rb85n\" (UniqueName: \"kubernetes.io/projected/8581921b-1c0a-4874-8bcf-b0fac6876a4c-kube-api-access-rb85n\") pod \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") "
Nov 03 20:44:39.652883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.652729    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-home\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-home\") pod \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") "
Nov 03 20:44:39.652883 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.652788    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-model-cache\") pod \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\" (UID: \"8581921b-1c0a-4874-8bcf-b0fac6876a4c\") "
Nov 03 20:44:39.653856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.653288    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tmp-dir" pod "8581921b-1c0a-4874-8bcf-b0fac6876a4c" (UID: "8581921b-1c0a-4874-8bcf-b0fac6876a4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:44:39.653856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.653362    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-model-cache" pod "8581921b-1c0a-4874-8bcf-b0fac6876a4c" (UID: "8581921b-1c0a-4874-8bcf-b0fac6876a4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:44:39.653856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.653823    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-home" pod "8581921b-1c0a-4874-8bcf-b0fac6876a4c" (UID: "8581921b-1c0a-4874-8bcf-b0fac6876a4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:44:39.656099 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.656060    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-dshm" pod "8581921b-1c0a-4874-8bcf-b0fac6876a4c" (UID: "8581921b-1c0a-4874-8bcf-b0fac6876a4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:44:39.656351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.656327    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8581921b-1c0a-4874-8bcf-b0fac6876a4c-kube-api-access-rb85n" pod "8581921b-1c0a-4874-8bcf-b0fac6876a4c" (UID: "8581921b-1c0a-4874-8bcf-b0fac6876a4c"). InnerVolumeSpecName "kube-api-access-rb85n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:44:39.656434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.656397    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tls-certs" pod "8581921b-1c0a-4874-8bcf-b0fac6876a4c" (UID: "8581921b-1c0a-4874-8bcf-b0fac6876a4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:44:39.657579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.657559    5282 scope.go:122] "RemoveContainer" containerID="f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238"
Nov 03 20:44:39.657853 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:44:39.657824    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238\": container with ID starting with f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238 not found: ID does not exist" containerID="f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238"
Nov 03 20:44:39.657928 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.657860    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238"} err="failed to get container status \"f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238\": rpc error: code = NotFound desc = could not find container \"f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238\": container with ID starting with f9c4e6c59676062fa68c7a6495c0a7b6633c401f6a3ec6a84fa15d58af4ef238 not found: ID does not exist"
Nov 03 20:44:39.753630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.753541    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb85n\" (UniqueName: \"kubernetes.io/projected/8581921b-1c0a-4874-8bcf-b0fac6876a4c-kube-api-access-rb85n\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:44:39.753630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.753574    5282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-home\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:44:39.753630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.753584    5282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-model-cache\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:44:39.753630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.753593    5282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-dshm\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:44:39.753630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.753602    5282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tls-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:44:39.753630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.753609    5282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8581921b-1c0a-4874-8bcf-b0fac6876a4c-tmp-dir\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:44:39.967842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.967802    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"]
Nov 03 20:44:39.972405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:39.972378    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq"]
Nov 03 20:44:40.347303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:40.344434    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-c8e4e36f-kserve-5b8b559c45-kx7rq" podUID="8581921b-1c0a-4874-8bcf-b0fac6876a4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.207:8000/ready\": context deadline exceeded"
Nov 03 20:44:41.457914 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:41.457511    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8581921b-1c0a-4874-8bcf-b0fac6876a4c" path="/var/lib/kubelet/pods/8581921b-1c0a-4874-8bcf-b0fac6876a4c/volumes"
Nov 03 20:44:42.731247 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:42.730307    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:44:43.906396 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:43.906356    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-krhrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:44:43.906990 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:43.906952    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-krhrv" podUID="f0ac7456-d600-48bd-b83d-8ed330bf70c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:44:59.961323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:44:59.961239    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:45:01.356788 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.356756    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-876c878b5-njv6s"]
Nov 03 20:45:01.357170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.357147    5282 cpu_manager.go:414] "containerMap: removing container" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy"
Nov 03 20:45:01.357170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.357159    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e5115a-f841-460f-83b5-7800a6ac4f01" containerName="istio-proxy"
Nov 03 20:45:01.357268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.357174    5282 cpu_manager.go:414] "containerMap: removing container" podUID="8581921b-1c0a-4874-8bcf-b0fac6876a4c" containerName="main"
Nov 03 20:45:01.357268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.357179    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8581921b-1c0a-4874-8bcf-b0fac6876a4c" containerName="main"
Nov 03 20:45:01.357340 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.357283    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8581921b-1c0a-4874-8bcf-b0fac6876a4c" containerName="main"
Nov 03 20:45:01.360460 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.360443    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.371030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.371002    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-876c878b5-njv6s"]
Nov 03 20:45:01.506745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.506708    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/324d125b-1199-4c96-ba01-2928290bd3f8-maas-api-tls\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.506931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.506764    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/324d125b-1199-4c96-ba01-2928290bd3f8-metrics-certs\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.506931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.506872    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/324d125b-1199-4c96-ba01-2928290bd3f8-openshift-service-ca\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.507054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.506929    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46zb\" (UniqueName: \"kubernetes.io/projected/324d125b-1199-4c96-ba01-2928290bd3f8-kube-api-access-z46zb\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.607652 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.607564    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/324d125b-1199-4c96-ba01-2928290bd3f8-maas-api-tls\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.607839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.607777    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/324d125b-1199-4c96-ba01-2928290bd3f8-metrics-certs\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.607839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.607833    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/324d125b-1199-4c96-ba01-2928290bd3f8-openshift-service-ca\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.607971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.607924    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46zb\" (UniqueName: \"kubernetes.io/projected/324d125b-1199-4c96-ba01-2928290bd3f8-kube-api-access-z46zb\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.608820 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.608786    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/324d125b-1199-4c96-ba01-2928290bd3f8-openshift-service-ca\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.610052 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.610027    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/324d125b-1199-4c96-ba01-2928290bd3f8-maas-api-tls\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.610239 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.610220    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/324d125b-1199-4c96-ba01-2928290bd3f8-metrics-certs\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.615217 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:45:01.615170    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:45:01.615627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.615604    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46zb\" (UniqueName: \"kubernetes.io/projected/324d125b-1199-4c96-ba01-2928290bd3f8-kube-api-access-z46zb\") pod \"maas-api-876c878b5-njv6s\" (UID: \"324d125b-1199-4c96-ba01-2928290bd3f8\") " pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.674632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.674596    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:01.948499 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.948408    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-876c878b5-njv6s"]
Nov 03 20:45:01.951164 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:45:01.951132    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324d125b_1199_4c96_ba01_2928290bd3f8.slice/crio-bf5adad87ad6473e37aa51640d7af43bf7f21346f23783591eec8e93b0d619c0 WatchSource:0}: Error finding container bf5adad87ad6473e37aa51640d7af43bf7f21346f23783591eec8e93b0d619c0: Status 404 returned error can't find the container with id bf5adad87ad6473e37aa51640d7af43bf7f21346f23783591eec8e93b0d619c0
Nov 03 20:45:01.952540 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:01.952523    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:45:02.781993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:02.781958    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s" event={"ID":"324d125b-1199-4c96-ba01-2928290bd3f8","Type":"ContainerStarted","Data":"c51815d7b5063b130be039b0af501e22d3a495931d55a14aed235500e7d51855"}
Nov 03 20:45:02.782434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:02.782013    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s" event={"ID":"324d125b-1199-4c96-ba01-2928290bd3f8","Type":"ContainerStarted","Data":"bf5adad87ad6473e37aa51640d7af43bf7f21346f23783591eec8e93b0d619c0"}
Nov 03 20:45:02.782434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:02.782055    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:02.799739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:02.799684    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s" podStartSLOduration=1.33029667 podStartE2EDuration="1.799668566s" totalImagesPullingTime="469.371896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.798885094 +0000 UTC m=+3895.468086563" watchObservedRunningTime="2026-11-03 20:45:02.799668566 +0000 UTC m=+3895.468870037"
Nov 03 20:45:08.276459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:08.276422    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:08.715581 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:08.715498    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:08.791727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:08.791699    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-876c878b5-njv6s"
Nov 03 20:45:08.842586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:08.842540    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"]
Nov 03 20:45:08.842870 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:08.842837    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g" podUID="0ce07bb7-3be0-4e7c-83ce-e12206831846" containerName="maas-api" containerID="cri-o://651f253221af677ca3e48f5397600e4ced1b467e1247d07acb44ac27c5304d32" gracePeriod=30
Nov 03 20:45:08.854009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:08.853980    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:10.181698 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.181663    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 20:45:10.286747 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.286712    5282 scope.go:122] "RemoveContainer" containerID="651f253221af677ca3e48f5397600e4ced1b467e1247d07acb44ac27c5304d32"
Nov 03 20:45:10.296339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.296316    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce07bb7-3be0-4e7c-83ce-e12206831846-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce07bb7-3be0-4e7c-83ce-e12206831846-openshift-service-ca\") pod \"0ce07bb7-3be0-4e7c-83ce-e12206831846\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") "
Nov 03 20:45:10.296481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.296381    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-metrics-certs\") pod \"0ce07bb7-3be0-4e7c-83ce-e12206831846\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") "
Nov 03 20:45:10.296481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.296480    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-maas-api-tls\") pod \"0ce07bb7-3be0-4e7c-83ce-e12206831846\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") "
Nov 03 20:45:10.296614 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.296533    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce07bb7-3be0-4e7c-83ce-e12206831846-kube-api-access-q5fls\" (UniqueName: \"kubernetes.io/projected/0ce07bb7-3be0-4e7c-83ce-e12206831846-kube-api-access-q5fls\") pod \"0ce07bb7-3be0-4e7c-83ce-e12206831846\" (UID: \"0ce07bb7-3be0-4e7c-83ce-e12206831846\") "
Nov 03 20:45:10.296847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.296753    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce07bb7-3be0-4e7c-83ce-e12206831846-openshift-service-ca" pod "0ce07bb7-3be0-4e7c-83ce-e12206831846" (UID: "0ce07bb7-3be0-4e7c-83ce-e12206831846"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:10.298663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.298633    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce07bb7-3be0-4e7c-83ce-e12206831846-kube-api-access-q5fls" pod "0ce07bb7-3be0-4e7c-83ce-e12206831846" (UID: "0ce07bb7-3be0-4e7c-83ce-e12206831846"). InnerVolumeSpecName "kube-api-access-q5fls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:10.298663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.298633    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-maas-api-tls" pod "0ce07bb7-3be0-4e7c-83ce-e12206831846" (UID: "0ce07bb7-3be0-4e7c-83ce-e12206831846"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:10.299304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.299284    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-metrics-certs" pod "0ce07bb7-3be0-4e7c-83ce-e12206831846" (UID: "0ce07bb7-3be0-4e7c-83ce-e12206831846"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:10.398136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.398105    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:10.398136 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.398134    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5fls\" (UniqueName: \"kubernetes.io/projected/0ce07bb7-3be0-4e7c-83ce-e12206831846-kube-api-access-q5fls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:10.398366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.398146    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce07bb7-3be0-4e7c-83ce-e12206831846-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:10.398366 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.398155    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ce07bb7-3be0-4e7c-83ce-e12206831846-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:10.822365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.822325    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g" event={"ID":"0ce07bb7-3be0-4e7c-83ce-e12206831846","Type":"ContainerDied","Data":"651f253221af677ca3e48f5397600e4ced1b467e1247d07acb44ac27c5304d32"}
Nov 03 20:45:10.822365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.822353    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"
Nov 03 20:45:10.822591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.822396    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g" event={"ID":"0ce07bb7-3be0-4e7c-83ce-e12206831846","Type":"ContainerDied","Data":"c61154b1748c22a9c08ebfe27322a9566a7f473084b502c26a7cb0899043e68d"}
Nov 03 20:45:10.861589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.861555    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"]
Nov 03 20:45:10.868056 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:10.868027    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb65579d5-gj67g"]
Nov 03 20:45:11.127257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:11.127176    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:11.259388 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:11.259351    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:11.453088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:11.453013    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce07bb7-3be0-4e7c-83ce-e12206831846" path="/var/lib/kubelet/pods/0ce07bb7-3be0-4e7c-83ce-e12206831846/volumes"
Nov 03 20:45:12.269216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.269168    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"]
Nov 03 20:45:12.269738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.269714    5282 cpu_manager.go:414] "containerMap: removing container" podUID="0ce07bb7-3be0-4e7c-83ce-e12206831846" containerName="maas-api"
Nov 03 20:45:12.269738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.269739    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce07bb7-3be0-4e7c-83ce-e12206831846" containerName="maas-api"
Nov 03 20:45:12.269960 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.269868    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce07bb7-3be0-4e7c-83ce-e12206831846" containerName="maas-api"
Nov 03 20:45:12.274726 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.274704    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.281505 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.281477    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-364642e1-serving-cert\""
Nov 03 20:45:12.284856 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.284644    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"]
Nov 03 20:45:12.303553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.303520    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:12.332636 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.332603    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"]
Nov 03 20:45:12.336626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.336604    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.348356 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.348329    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-364642e1\""
Nov 03 20:45:12.349808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.349785    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"]
Nov 03 20:45:12.358361 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.358277    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-364642e1-dockercfg-dfhvc\""
Nov 03 20:45:12.382793 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.382752    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"]
Nov 03 20:45:12.386405 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.386387    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.396369 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.396346    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"]
Nov 03 20:45:12.418777 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.418740    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-openshift-service-ca\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.418918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.418811    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-maas-api-tls\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.418918 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.418863    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2sc\" (UniqueName: \"kubernetes.io/projected/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-kube-api-access-lr2sc\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.418993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.418930    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-metrics-certs\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.446470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.446435    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:12.519905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.519797    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-openshift-service-ca\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.520101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.519867    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82x9\" (UniqueName: \"kubernetes.io/projected/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-kube-api-access-j82x9\") pod \"payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.520101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.519984    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-maas-api-tls\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.520101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.520037    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") " pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.520101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.520068    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.520101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.520096    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2sc\" (UniqueName: \"kubernetes.io/projected/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-kube-api-access-lr2sc\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.520474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.520117    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-metrics-certs\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.520474 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.520182    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2mn\" (UniqueName: \"kubernetes.io/projected/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-kube-api-access-wt2mn\") pod \"payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") " pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.520643 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.520619    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-openshift-service-ca\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.523098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.523072    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-maas-api-tls\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.523341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.523322    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-metrics-certs\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.528598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.528563    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2sc\" (UniqueName: \"kubernetes.io/projected/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-kube-api-access-lr2sc\") pod \"maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.557493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.557459    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:12.593451 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.593405    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:12.621101 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.621071    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82x9\" (UniqueName: \"kubernetes.io/projected/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-kube-api-access-j82x9\") pod \"payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.621266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.621122    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") " pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.621266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.621142    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.621266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.621188    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2mn\" (UniqueName: \"kubernetes.io/projected/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-kube-api-access-wt2mn\") pod \"payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") " pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.621926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.621898    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.622397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.621898    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-plugins-config-volume\") pod \"payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") " pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.629680 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.629653    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82x9\" (UniqueName: \"kubernetes.io/projected/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-kube-api-access-j82x9\") pod \"payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.629995 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.629976    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2mn\" (UniqueName: \"kubernetes.io/projected/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-kube-api-access-wt2mn\") pod \"payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") " pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.651183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.651148    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:12.700863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.700825    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:12.871845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.871804    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"]
Nov 03 20:45:12.874190 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:45:12.874159    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcde640_107c_4e46_9bc7_b6f9ab68fa9d.slice/crio-88b517b9e3acc4f62bb558506f3523fab4ee0f8fa5102bdf37fa48c4d3ce3f7c WatchSource:0}: Error finding container 88b517b9e3acc4f62bb558506f3523fab4ee0f8fa5102bdf37fa48c4d3ce3f7c: Status 404 returned error can't find the container with id 88b517b9e3acc4f62bb558506f3523fab4ee0f8fa5102bdf37fa48c4d3ce3f7c
Nov 03 20:45:12.937900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.937861    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"]
Nov 03 20:45:12.941745 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:45:12.941715    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19fc33f_7054_42e5_b69e_a2998cbe4e5c.slice/crio-65da8b52ec6d477d7183cb6650c4a60356641104391295453c1fa22a8ab1eda1 WatchSource:0}: Error finding container 65da8b52ec6d477d7183cb6650c4a60356641104391295453c1fa22a8ab1eda1: Status 404 returned error can't find the container with id 65da8b52ec6d477d7183cb6650c4a60356641104391295453c1fa22a8ab1eda1
Nov 03 20:45:12.978996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:12.978956    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"]
Nov 03 20:45:12.982773 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:45:12.982745    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa86ce97_7034_4577_a0d5_d8f31a6dd6e2.slice/crio-6c22a2da4d3228c669faca5591c08ec146ee7178c389677fe751c2fb4d9bed13 WatchSource:0}: Error finding container 6c22a2da4d3228c669faca5591c08ec146ee7178c389677fe751c2fb4d9bed13: Status 404 returned error can't find the container with id 6c22a2da4d3228c669faca5591c08ec146ee7178c389677fe751c2fb4d9bed13
Nov 03 20:45:13.148022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.147923    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:13.842632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.842596    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx" event={"ID":"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d","Type":"ContainerStarted","Data":"bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c"}
Nov 03 20:45:13.843143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.842660    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx" event={"ID":"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d","Type":"ContainerStarted","Data":"88b517b9e3acc4f62bb558506f3523fab4ee0f8fa5102bdf37fa48c4d3ce3f7c"}
Nov 03 20:45:13.843143 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.842686    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:13.844287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.844259    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm" event={"ID":"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2","Type":"ContainerStarted","Data":"615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede"}
Nov 03 20:45:13.844456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.844318    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:13.844456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.844345    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm" event={"ID":"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2","Type":"ContainerStarted","Data":"6c22a2da4d3228c669faca5591c08ec146ee7178c389677fe751c2fb4d9bed13"}
Nov 03 20:45:13.845673 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.845647    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp" event={"ID":"a19fc33f-7054-42e5-b69e-a2998cbe4e5c","Type":"ContainerStarted","Data":"0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5"}
Nov 03 20:45:13.845785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.845704    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp" event={"ID":"a19fc33f-7054-42e5-b69e-a2998cbe4e5c","Type":"ContainerStarted","Data":"65da8b52ec6d477d7183cb6650c4a60356641104391295453c1fa22a8ab1eda1"}
Nov 03 20:45:13.845785 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.845777    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:13.860291 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.860246    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx" podStartSLOduration=1.451431925 podStartE2EDuration="1.860228109s" totalImagesPullingTime="408.796184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:13.858901664 +0000 UTC m=+3906.528103136" watchObservedRunningTime="2026-11-03 20:45:13.860228109 +0000 UTC m=+3906.529429573"
Nov 03 20:45:13.873621 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.873577    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm" podStartSLOduration=1.8735659820000001 podStartE2EDuration="1.873565982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:13.872865861 +0000 UTC m=+3906.542067345" watchObservedRunningTime="2026-11-03 20:45:13.873565982 +0000 UTC m=+3906.542767468"
Nov 03 20:45:13.888267 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.888189    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp" podStartSLOduration=1.888177121 podStartE2EDuration="1.888177121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:13.886761377 +0000 UTC m=+3906.555962848" watchObservedRunningTime="2026-11-03 20:45:13.888177121 +0000 UTC m=+3906.557378591"
Nov 03 20:45:13.912011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:13.911972    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:14.523257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:14.523216    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:45:14.841088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:14.841002    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:15.513900 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:15.513867    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:16.119294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:16.119258    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:16.739303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:16.739270    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:17.431688 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:17.431570    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:18.023114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:18.023081    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:18.626401 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:18.626363    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:19.314761 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:19.314731    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:19.630819 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:19.630732    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:19.753170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:19.753140    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:19.867718 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:19.867682    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:19.871833 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:19.871801    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:20.450000 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:20.449968    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:21.303606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:21.303565    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:21.913845 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:21.913811    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:22.537812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:22.537772    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:22.974134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:22.974030    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:23.313183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:23.313132    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:23.921296 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:23.921264    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:24.689766 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:24.689734    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:24.856744 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:24.856709    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:24.857029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:24.857005    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:25.331884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:25.331851    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:25.936978 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:25.936946    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:26.603797 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:26.603762    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:27.407431 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:27.407401    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:28.045987 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:28.045948    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:28.912501 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:28.912455    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:29.518758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:29.518726    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:30.178182 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:30.178148    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:30.779109 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:30.779070    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:31.429446 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:31.429413    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:31.613519 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:45:31.613483    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:45:32.113898 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:32.113865    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:32.732679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:32.732647    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:33.542557 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:33.542511    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:34.257303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:34.257262    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:34.956393 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:34.956350    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:35.640853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:35.640819    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:36.249770 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:36.249725    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:37.055923 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:37.055884    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"]
Nov 03 20:45:37.070970 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:37.070929    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"]
Nov 03 20:45:37.972465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:37.972420    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"]
Nov 03 20:45:37.972738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:37.972685    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx" podUID="5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" containerName="maas-api" containerID="cri-o://bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c" gracePeriod=30
Nov 03 20:45:38.076003 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.075970    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:38.197714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.196800    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:38.257257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.257230    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:38.393677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.393307    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:38.421745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.421711    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-openshift-service-ca\") pod \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") "
Nov 03 20:45:38.421950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.421805    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-maas-api-tls\") pod \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") "
Nov 03 20:45:38.421950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.421832    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-metrics-certs\") pod \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") "
Nov 03 20:45:38.421950 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.421877    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-kube-api-access-lr2sc\" (UniqueName: \"kubernetes.io/projected/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-kube-api-access-lr2sc\") pod \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\" (UID: \"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d\") "
Nov 03 20:45:38.422230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.422188    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-openshift-service-ca" pod "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" (UID: "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:38.424025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.424002    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-metrics-certs" pod "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" (UID: "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:38.424025 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.424011    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-kube-api-access-lr2sc" pod "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" (UID: "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d"). InnerVolumeSpecName "kube-api-access-lr2sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:38.424221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.424031    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-maas-api-tls" pod "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" (UID: "5fcde640-107c-4e46-9bc7-b6f9ab68fa9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:38.523559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.523461    5282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-openshift-service-ca\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:38.523559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.523503    5282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-maas-api-tls\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:38.523559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.523519    5282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-metrics-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:38.523559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.523533    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2sc\" (UniqueName: \"kubernetes.io/projected/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d-kube-api-access-lr2sc\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:38.976933 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.976900    5282 generic.go:356] "Generic (PLEG): container finished" podID="5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" containerID="bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c" exitCode=0
Nov 03 20:45:38.977098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.976951    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"
Nov 03 20:45:38.977098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.976988    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx" event={"ID":"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d","Type":"ContainerDied","Data":"bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c"}
Nov 03 20:45:38.977098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.977043    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx" event={"ID":"5fcde640-107c-4e46-9bc7-b6f9ab68fa9d","Type":"ContainerDied","Data":"88b517b9e3acc4f62bb558506f3523fab4ee0f8fa5102bdf37fa48c4d3ce3f7c"}
Nov 03 20:45:38.977098 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.977075    5282 scope.go:122] "RemoveContainer" containerID="bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c"
Nov 03 20:45:38.986852 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.986832    5282 scope.go:122] "RemoveContainer" containerID="bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c"
Nov 03 20:45:38.987123 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:45:38.987101    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c\": container with ID starting with bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c not found: ID does not exist" containerID="bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c"
Nov 03 20:45:38.987167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.987132    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c"} err="failed to get container status \"bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c\": rpc error: code = NotFound desc = could not find container \"bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c\": container with ID starting with bb1c4d77ff122712d8d36c0676a09899350c189a8f1fe3bb87eb67d62db3622c not found: ID does not exist"
Nov 03 20:45:38.999287 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:38.999265    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"]
Nov 03 20:45:39.003982 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:39.003946    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-364642e1-89fdb5bdb-5v4kx"]
Nov 03 20:45:39.453086 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:39.453007    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" path="/var/lib/kubelet/pods/5fcde640-107c-4e46-9bc7-b6f9ab68fa9d/volumes"
Nov 03 20:45:39.747868 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:39.747833    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:45:42.057013 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.056972    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm" podUID="aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" containerName="payload-processing" containerID="cri-o://615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede" gracePeriod=25
Nov 03 20:45:42.071276 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.071247    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp" podUID="a19fc33f-7054-42e5-b69e-a2998cbe4e5c" containerName="payload-pre-processing" containerID="cri-o://0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5" gracePeriod=25
Nov 03 20:45:42.304365 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.304344    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:42.334454 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.334428    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:42.363137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.363110    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-plugins-config-volume\") pod \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") "
Nov 03 20:45:42.363304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.363182    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-kube-api-access-wt2mn\" (UniqueName: \"kubernetes.io/projected/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-kube-api-access-wt2mn\") pod \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\" (UID: \"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2\") "
Nov 03 20:45:42.363377 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.363355    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-plugins-config-volume\") pod \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") "
Nov 03 20:45:42.363473 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.363395    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-kube-api-access-j82x9\" (UniqueName: \"kubernetes.io/projected/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-kube-api-access-j82x9\") pod \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\" (UID: \"a19fc33f-7054-42e5-b69e-a2998cbe4e5c\") "
Nov 03 20:45:42.363583 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.363483    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-plugins-config-volume" pod "aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" (UID: "aa86ce97-7034-4577-a0d5-d8f31a6dd6e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:42.363732 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.363709    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:42.363814 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.363772    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-plugins-config-volume" pod "a19fc33f-7054-42e5-b69e-a2998cbe4e5c" (UID: "a19fc33f-7054-42e5-b69e-a2998cbe4e5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:42.365442 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.365412    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-kube-api-access-wt2mn" pod "aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" (UID: "aa86ce97-7034-4577-a0d5-d8f31a6dd6e2"). InnerVolumeSpecName "kube-api-access-wt2mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:42.365523 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.365441    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-kube-api-access-j82x9" pod "a19fc33f-7054-42e5-b69e-a2998cbe4e5c" (UID: "a19fc33f-7054-42e5-b69e-a2998cbe4e5c"). InnerVolumeSpecName "kube-api-access-j82x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:42.465046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.465011    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2mn\" (UniqueName: \"kubernetes.io/projected/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2-kube-api-access-wt2mn\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:42.465046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.465041    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:42.465046 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.465051    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j82x9\" (UniqueName: \"kubernetes.io/projected/a19fc33f-7054-42e5-b69e-a2998cbe4e5c-kube-api-access-j82x9\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:42.997413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.997376    5282 generic.go:356] "Generic (PLEG): container finished" podID="aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" containerID="615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede" exitCode=0
Nov 03 20:45:42.997647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.997420    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"
Nov 03 20:45:42.997647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.997464    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm" event={"ID":"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2","Type":"ContainerDied","Data":"615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede"}
Nov 03 20:45:42.997647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.997543    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm" event={"ID":"aa86ce97-7034-4577-a0d5-d8f31a6dd6e2","Type":"ContainerDied","Data":"6c22a2da4d3228c669faca5591c08ec146ee7178c389677fe751c2fb4d9bed13"}
Nov 03 20:45:42.997647 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.997574    5282 scope.go:122] "RemoveContainer" containerID="615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede"
Nov 03 20:45:42.998738 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.998712    5282 generic.go:356] "Generic (PLEG): container finished" podID="a19fc33f-7054-42e5-b69e-a2998cbe4e5c" containerID="0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5" exitCode=0
Nov 03 20:45:42.998834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.998748    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp" event={"ID":"a19fc33f-7054-42e5-b69e-a2998cbe4e5c","Type":"ContainerDied","Data":"0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5"}
Nov 03 20:45:42.998834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.998765    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"
Nov 03 20:45:42.998834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:42.998796    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp" event={"ID":"a19fc33f-7054-42e5-b69e-a2998cbe4e5c","Type":"ContainerDied","Data":"65da8b52ec6d477d7183cb6650c4a60356641104391295453c1fa22a8ab1eda1"}
Nov 03 20:45:43.006282 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.006262    5282 scope.go:122] "RemoveContainer" containerID="615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede"
Nov 03 20:45:43.006552 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:45:43.006529    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede\": container with ID starting with 615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede not found: ID does not exist" containerID="615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede"
Nov 03 20:45:43.006667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.006561    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede"} err="failed to get container status \"615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede\": rpc error: code = NotFound desc = could not find container \"615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede\": container with ID starting with 615d67ea65ca59ed185fc60cea69d4064dec498fc4be4bb2c4a96073638e0ede not found: ID does not exist"
Nov 03 20:45:43.006667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.006585    5282 scope.go:122] "RemoveContainer" containerID="0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5"
Nov 03 20:45:43.014293 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.014277    5282 scope.go:122] "RemoveContainer" containerID="0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5"
Nov 03 20:45:43.014525 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:45:43.014505    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5\": container with ID starting with 0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5 not found: ID does not exist" containerID="0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5"
Nov 03 20:45:43.014590 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.014535    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5"} err="failed to get container status \"0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5\": rpc error: code = NotFound desc = could not find container \"0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5\": container with ID starting with 0e121fba5fcf8631d1d142700567abc8b8b87d6d876cd2603720c78526ff35e5 not found: ID does not exist"
Nov 03 20:45:43.022309 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.022288    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"]
Nov 03 20:45:43.026310 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.026288    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-364642e1-868b5cc9cb-7vrmm"]
Nov 03 20:45:43.044694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.044670    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"]
Nov 03 20:45:43.047119 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.047091    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-364642e1-6794bdd4dc-gcpsp"]
Nov 03 20:45:43.401412 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.401330    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:43.457399 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.457366    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19fc33f-7054-42e5-b69e-a2998cbe4e5c" path="/var/lib/kubelet/pods/a19fc33f-7054-42e5-b69e-a2998cbe4e5c/volumes"
Nov 03 20:45:43.457821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:43.457806    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" path="/var/lib/kubelet/pods/aa86ce97-7034-4577-a0d5-d8f31a6dd6e2/volumes"
Nov 03 20:45:44.688066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:44.688037    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:45.050083 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050043    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"]
Nov 03 20:45:45.050781 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050757    5282 cpu_manager.go:414] "containerMap: removing container" podUID="5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" containerName="maas-api"
Nov 03 20:45:45.050841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050787    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" containerName="maas-api"
Nov 03 20:45:45.050889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050849    5282 cpu_manager.go:414] "containerMap: removing container" podUID="aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" containerName="payload-processing"
Nov 03 20:45:45.050889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050859    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" containerName="payload-processing"
Nov 03 20:45:45.050889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050871    5282 cpu_manager.go:414] "containerMap: removing container" podUID="a19fc33f-7054-42e5-b69e-a2998cbe4e5c" containerName="payload-pre-processing"
Nov 03 20:45:45.050889 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050880    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19fc33f-7054-42e5-b69e-a2998cbe4e5c" containerName="payload-pre-processing"
Nov 03 20:45:45.051018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.050982    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcde640-107c-4e46-9bc7-b6f9ab68fa9d" containerName="maas-api"
Nov 03 20:45:45.051018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.051006    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa86ce97-7034-4577-a0d5-d8f31a6dd6e2" containerName="payload-processing"
Nov 03 20:45:45.051018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.051015    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19fc33f-7054-42e5-b69e-a2998cbe4e5c" containerName="payload-pre-processing"
Nov 03 20:45:45.055022 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.054998    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.059667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.059639    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-ca53c0-openshift-default-dockercfg-wcpkp\""
Nov 03 20:45:45.066568 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.065013    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"]
Nov 03 20:45:45.084591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.084562    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-data\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.084725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.084620    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istiod-ca-cert\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.084725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.084707    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-podinfo\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.084843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.084735    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-token\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.084843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.084760    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-certs\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.084963 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.084884    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966qf\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-kube-api-access-966qf\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.085023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.084966    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-credential-socket\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.085023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.085012    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-envoy\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.085137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.085096    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-socket\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.127336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.127279    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:45.186035 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186005    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-credential-socket\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186057    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-envoy\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186105    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-socket\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186134    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-data\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186189    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istiod-ca-cert\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186252    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-podinfo\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186292    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-token\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186324    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-certs\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186369    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966qf\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-kube-api-access-966qf\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186453    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-credential-socket\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186557    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-data\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186729 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186602    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-socket\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.186827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.186811    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-certs\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.187120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.187096    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istiod-ca-cert\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.188380 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.188361    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-envoy\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.188783 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.188761    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-podinfo\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.195727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.195699    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-token\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.196303 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.196256    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-966qf\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-kube-api-access-966qf\") pod \"e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") " pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.258685 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.258527    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:45.370699 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.370610    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:45.650316 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.650230    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"]
Nov 03 20:45:45.653229 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:45:45.653171    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f0fe09_4261_4779_af7b_10ccf9dd1bf4.slice/crio-e92b3aefdf3f4e92a38818f52197c22c8a3507f6969ebed009a10bf3d8b1bca4 WatchSource:0}: Error finding container e92b3aefdf3f4e92a38818f52197c22c8a3507f6969ebed009a10bf3d8b1bca4: Status 404 returned error can't find the container with id e92b3aefdf3f4e92a38818f52197c22c8a3507f6969ebed009a10bf3d8b1bca4
Nov 03 20:45:45.655339 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.655307    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:45:45.655415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.655373    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:45:45.655415 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:45.655402    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:45:46.023795 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:46.023760    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" event={"ID":"26f0fe09-4261-4779-af7b-10ccf9dd1bf4","Type":"ContainerStarted","Data":"bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234"}
Nov 03 20:45:46.024255 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:46.023812    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" event={"ID":"26f0fe09-4261-4779-af7b-10ccf9dd1bf4","Type":"ContainerStarted","Data":"e92b3aefdf3f4e92a38818f52197c22c8a3507f6969ebed009a10bf3d8b1bca4"}
Nov 03 20:45:46.043854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:46.043795    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podStartSLOduration=1.043778504 podStartE2EDuration="1.043778504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:46.043388246 +0000 UTC m=+3938.712589751" watchObservedRunningTime="2026-11-03 20:45:46.043778504 +0000 UTC m=+3938.712979971"
Nov 03 20:45:46.371644 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:46.371552    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:46.376736 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:46.376711    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:47.028649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:47.028619    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:47.029500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:47.029476    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:45:48.240847 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:48.240812    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:48.371054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:48.371018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:49.434721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.434689    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:49.469953 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.469913    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"]
Nov 03 20:45:49.472931 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.472913    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.482234 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.482186    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-ca53c0\""
Nov 03 20:45:49.482428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.482288    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-ca53c0-dockercfg-jhbxp\""
Nov 03 20:45:49.483875 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.483854    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"]
Nov 03 20:45:49.506187 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.506154    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"]
Nov 03 20:45:49.509683 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.509664    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:49.517748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.517726    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"]
Nov 03 20:45:49.525957 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.525935    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c920d708-0527-417f-88c0-c10fa9a7bb0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.526075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.525965    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjkq\" (UniqueName: \"kubernetes.io/projected/c920d708-0527-417f-88c0-c10fa9a7bb0b-kube-api-access-kzjkq\") pod \"payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.526075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.526004    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:49.526190 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.526120    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sgv\" (UniqueName: \"kubernetes.io/projected/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-kube-api-access-d4sgv\") pod \"payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:49.566709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.566682    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:49.627048 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.627000    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjkq\" (UniqueName: \"kubernetes.io/projected/c920d708-0527-417f-88c0-c10fa9a7bb0b-kube-api-access-kzjkq\") pod \"payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.627265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.627099    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:49.627265 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.627147    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sgv\" (UniqueName: \"kubernetes.io/projected/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-kube-api-access-d4sgv\") pod \"payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:49.627394 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.627288    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c920d708-0527-417f-88c0-c10fa9a7bb0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.628231 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.628185    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c920d708-0527-417f-88c0-c10fa9a7bb0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.629330 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.629303    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:49.640180 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.640149    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjkq\" (UniqueName: \"kubernetes.io/projected/c920d708-0527-417f-88c0-c10fa9a7bb0b-kube-api-access-kzjkq\") pod \"payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.640648 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.640622    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sgv\" (UniqueName: \"kubernetes.io/projected/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-kube-api-access-d4sgv\") pod \"payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:49.699020 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.698960    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:49.787333 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.787299    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:49.824126 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:49.824093    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:50.067064 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:50.067035    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"]
Nov 03 20:45:50.069051 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:45:50.069023    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc920d708_0527_417f_88c0_c10fa9a7bb0b.slice/crio-600d1a8b948af4c165a5286d2dfae901ac4560ba75f35c438e4c716dc50c8bf3 WatchSource:0}: Error finding container 600d1a8b948af4c165a5286d2dfae901ac4560ba75f35c438e4c716dc50c8bf3: Status 404 returned error can't find the container with id 600d1a8b948af4c165a5286d2dfae901ac4560ba75f35c438e4c716dc50c8bf3
Nov 03 20:45:50.110257 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:50.110233    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"]
Nov 03 20:45:50.114382 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:45:50.114351    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c20692_0f49_4d2c_9a87_a672e7cc6e87.slice/crio-4568dfe69b8af6a58538a19e7b1a27bc293fc71edf72fb44f012a64ddb08a547 WatchSource:0}: Error finding container 4568dfe69b8af6a58538a19e7b1a27bc293fc71edf72fb44f012a64ddb08a547: Status 404 returned error can't find the container with id 4568dfe69b8af6a58538a19e7b1a27bc293fc71edf72fb44f012a64ddb08a547
Nov 03 20:45:50.354418 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:50.351412    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:51.048448 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.048394    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt" event={"ID":"c920d708-0527-417f-88c0-c10fa9a7bb0b","Type":"ContainerStarted","Data":"403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa"}
Nov 03 20:45:51.048959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.048466    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt" event={"ID":"c920d708-0527-417f-88c0-c10fa9a7bb0b","Type":"ContainerStarted","Data":"600d1a8b948af4c165a5286d2dfae901ac4560ba75f35c438e4c716dc50c8bf3"}
Nov 03 20:45:51.048959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.048521    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:45:51.050012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.049987    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq" event={"ID":"f5c20692-0f49-4d2c-9a87-a672e7cc6e87","Type":"ContainerStarted","Data":"b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28"}
Nov 03 20:45:51.050137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.050025    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq" event={"ID":"f5c20692-0f49-4d2c-9a87-a672e7cc6e87","Type":"ContainerStarted","Data":"4568dfe69b8af6a58538a19e7b1a27bc293fc71edf72fb44f012a64ddb08a547"}
Nov 03 20:45:51.050137 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.050114    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:45:51.066961 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.066913    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt" podStartSLOduration=2.066900243 podStartE2EDuration="2.066900243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:51.064810341 +0000 UTC m=+3943.734011811" watchObservedRunningTime="2026-11-03 20:45:51.066900243 +0000 UTC m=+3943.736101712"
Nov 03 20:45:51.081045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.081001    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq" podStartSLOduration=2.080990001 podStartE2EDuration="2.080990001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:51.080063317 +0000 UTC m=+3943.749264786" watchObservedRunningTime="2026-11-03 20:45:51.080990001 +0000 UTC m=+3943.750191495"
Nov 03 20:45:51.144274 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.144233    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:51.788598 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:51.788562    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:52.095392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:52.095310    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:45:52.454659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:52.454573    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:53.314834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:53.314797    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:54.038912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:54.038877    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:54.769021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:54.768984    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:55.646929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:55.646899    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:56.347558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:56.347523    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:57.053657 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:57.053625    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:57.814334 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:57.814298    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:58.638152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:58.638115    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:59.242029 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:59.241998    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:45:59.922897 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:45:59.922863    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:00.608040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:00.607994    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:01.349376 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:01.349329    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:01.612869 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:46:01.612790    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:46:01.913905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:01.913826    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:02.030150 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.030113    5282 patch_prober.go:29] interesting pod/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 20:46:02.030397 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.030221    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 20:46:02.058002 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.057972    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:46:02.058170 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.058035    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:46:02.150628 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.150585    5282 generic.go:356] "Generic (PLEG): container finished" podID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerID="bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234" exitCode=0
Nov 03 20:46:02.150804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.150647    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" event={"ID":"26f0fe09-4261-4779-af7b-10ccf9dd1bf4","Type":"ContainerDied","Data":"bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234"}
Nov 03 20:46:02.151042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.151026    5282 scope.go:122] "RemoveContainer" containerID="bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234"
Nov 03 20:46:02.152156 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.152127    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:46:02.152272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.152191    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:46:02.152272 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.152248    5282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 20:46:02.624682 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:02.624652    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:03.157125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:03.157086    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" event={"ID":"26f0fe09-4261-4779-af7b-10ccf9dd1bf4","Type":"ContainerStarted","Data":"f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89"}
Nov 03 20:46:03.219535 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:03.219503    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:03.352410 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:03.352375    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:03.371865 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:03.371827    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:46:03.372337 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:03.372170    5282 patch_prober.go:29] interesting pod/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 20:46:03.372487 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:03.372373    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 20:46:04.153134 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:04.153100    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:04.371857 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:04.371825    5282 patch_prober.go:29] interesting pod/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 20:46:04.372441 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:04.372398    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 20:46:05.031023 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:05.030989    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:05.371486 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:05.371408    5282 patch_prober.go:29] interesting pod/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 20:46:05.371921 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:05.371460    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 20:46:05.758113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:05.758076    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:06.371758 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:06.371724    5282 patch_prober.go:29] interesting pod/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 20:46:06.372152 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:06.371787    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 20:46:06.436213 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:06.436112    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:07.135749 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:07.135712    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:07.371236 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:07.371171    5282 patch_prober.go:29] interesting pod/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 20:46:07.371467 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:07.371431    5282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 20:46:07.845863 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:07.845830    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:08.375687 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:08.375651    5282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:46:08.375956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:08.375931    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:46:08.376586 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:08.376564    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:46:08.549725 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:08.549689    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:09.226300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:09.226271    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:09.914237 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:09.914177    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:10.450315 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:10.450269    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:11.024624 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:11.024583    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:11.630746 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:11.630711    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:12.427667 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:12.427636    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:13.140684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:13.140650    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:13.621434 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:13.621397    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:14.222503 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:14.222474    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:14.970400 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:14.970344    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:15.515254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:15.515220    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:16.116707 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:16.116673    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:16.732182 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:16.732154    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:17.417351 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:17.417316    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:18.058050 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:18.057963    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:18.728943 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:18.728896    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:19.332481 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:19.332445    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:19.936811 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:19.936775    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:20.623385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:20.623357    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:20.852894 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:20.852863    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:21.243935 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:21.243899    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:21.837837 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:21.837803    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:22.518011 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:22.517974    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:23.011668 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:23.011628    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:23.231959 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:23.231920    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:23.860221 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:23.859932    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:24.439207 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:24.439164    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:25.241275 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:25.241239    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:25.934890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:25.934849    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:26.658249 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:26.658186    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:27.322975 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:27.322945    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:27.938494 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:27.938461    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:28.672822 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:28.671949    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:29.349063 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:29.349031    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:30.032878 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:30.032838    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:30.627009 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:30.626971    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:31.479755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:31.479708    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:31.613538 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:46:31.613504    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:46:32.014367 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:32.014336    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:32.667314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:32.667265    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:33.325932 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:33.325898    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:33.965677 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:33.965073    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:34.534186 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:34.534155    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:35.191317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:35.191279    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:35.841574 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:35.841537    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:36.527426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:36.527393    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:37.239447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:37.239411    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:38.019588 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:38.019554    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:38.848318 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:38.848281    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:39.624708 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:39.624673    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:40.250426 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:40.250390    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:40.933088 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:40.933053    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:41.628336 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:41.628295    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:42.249721 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:42.249626    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:42.947818 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:42.947771    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:43.623068 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:43.623036    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:44.281381 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:44.281350    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:44.918395 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:44.918361    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:45.765222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:45.765160    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:46.516536 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:46.516487    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:47.307560 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:47.307525    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:47.937489 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:47.937445    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:48.712183 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:48.712149    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:49.393864 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:49.393830    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:50.047701 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:50.047669    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:50.708059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:50.708025    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:51.380384 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:51.380344    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:52.051117 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:52.051087    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:52.633185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:52.633148    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:53.316579 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:53.316539    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:53.924601 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:53.924564    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:54.541408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:54.541373    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:55.229184 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:55.229151    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:55.841017 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:55.840986    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:56.544515 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:56.544443    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:57.259341 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:57.259290    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:57.928771 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:57.928725    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:58.614645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:58.614618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:46:59.378676 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:46:59.378631    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:00.033808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:00.033772    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:00.640480 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:00.640429    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:01.438605 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:01.438571    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:01.615137 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:47:01.615096    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:47:02.016100 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:02.016058    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:02.628659 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:02.628618    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:03.312946 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:03.312912    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:04.018268 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:04.018234    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:04.748014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:04.747977    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:05.426877 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:05.426846    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:06.083001 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:06.082964    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:06.663379 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:06.663331    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:07.372471 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:07.372439    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:08.022458 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:08.022421    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:08.631511 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:08.631472    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:09.232694 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:09.232660    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:09.904660 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:09.904623    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:10.536756 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:10.536728    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:11.209470 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:11.209437    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:11.816956 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:11.816922    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:12.616465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:12.616431    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:13.216071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:13.216027    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:13.828222 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:13.828176    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:14.594416 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:14.594383    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:15.132606 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:15.132569    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:15.855705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:15.855643    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:16.436219 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:16.436167    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:17.059879 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:17.059835    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:17.627591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:17.627548    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:18.309663 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:18.309628    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:18.924596 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:18.924564    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:19.530359 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:19.530313    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:20.228686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:20.228627    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:20.818821 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:20.818789    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:21.507798 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:21.507763    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:22.116496 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:22.116460    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:22.727031 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:22.726995    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:23.069509 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:23.069415    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:23.520851 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:23.520816    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:24.332618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:24.332578    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:24.939626 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:24.939595    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:25.590189 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:25.590148    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:26.273867 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:26.273831    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:26.925787 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:26.925753    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:27.538679 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:27.538644    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:28.138637 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:28.138604    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:28.739630 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:28.739596    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:29.342459 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:29.342408    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:29.961841 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:29.961766    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:30.542831 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:30.542796    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:31.143993 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:31.143942    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:31.613940 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:47:31.613906    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:47:31.881043 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:31.880947    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:32.649054 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:32.648954    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:33.029292 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:33.029256    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"]
Nov 03 20:47:33.043801 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:33.043768    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"]
Nov 03 20:47:33.129929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:33.129895    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:33.252808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:33.252769    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:38.029691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.029647    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq" podUID="f5c20692-0f49-4d2c-9a87-a672e7cc6e87" containerName="payload-processing" containerID="cri-o://b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28" gracePeriod=25
Nov 03 20:47:38.044521 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.044473    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt" podUID="c920d708-0527-417f-88c0-c10fa9a7bb0b" containerName="payload-pre-processing" containerID="cri-o://403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa" gracePeriod=25
Nov 03 20:47:38.261059 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.261018    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:38.327151 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.327120    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:47:38.330181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.330157    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:47:38.406834 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.406795    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:38.473686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.472650    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-plugins-config-volume\") pod \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") "
Nov 03 20:47:38.473686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.472697    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c920d708-0527-417f-88c0-c10fa9a7bb0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c920d708-0527-417f-88c0-c10fa9a7bb0b-plugins-config-volume\") pod \"c920d708-0527-417f-88c0-c10fa9a7bb0b\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") "
Nov 03 20:47:38.473686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.472751    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-kube-api-access-d4sgv\" (UniqueName: \"kubernetes.io/projected/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-kube-api-access-d4sgv\") pod \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\" (UID: \"f5c20692-0f49-4d2c-9a87-a672e7cc6e87\") "
Nov 03 20:47:38.473686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.472859    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c920d708-0527-417f-88c0-c10fa9a7bb0b-kube-api-access-kzjkq\" (UniqueName: \"kubernetes.io/projected/c920d708-0527-417f-88c0-c10fa9a7bb0b-kube-api-access-kzjkq\") pod \"c920d708-0527-417f-88c0-c10fa9a7bb0b\" (UID: \"c920d708-0527-417f-88c0-c10fa9a7bb0b\") "
Nov 03 20:47:38.473686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.473011    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-plugins-config-volume" pod "f5c20692-0f49-4d2c-9a87-a672e7cc6e87" (UID: "f5c20692-0f49-4d2c-9a87-a672e7cc6e87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:38.473686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.473185    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:38.473686 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.473607    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c920d708-0527-417f-88c0-c10fa9a7bb0b-plugins-config-volume" pod "c920d708-0527-417f-88c0-c10fa9a7bb0b" (UID: "c920d708-0527-417f-88c0-c10fa9a7bb0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:38.492911 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.492378    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-kube-api-access-d4sgv" pod "f5c20692-0f49-4d2c-9a87-a672e7cc6e87" (UID: "f5c20692-0f49-4d2c-9a87-a672e7cc6e87"). InnerVolumeSpecName "kube-api-access-d4sgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:38.493589 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.493562    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c920d708-0527-417f-88c0-c10fa9a7bb0b-kube-api-access-kzjkq" pod "c920d708-0527-417f-88c0-c10fa9a7bb0b" (UID: "c920d708-0527-417f-88c0-c10fa9a7bb0b"). InnerVolumeSpecName "kube-api-access-kzjkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:38.518748 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.518711    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:38.574504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.574430    5282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c920d708-0527-417f-88c0-c10fa9a7bb0b-plugins-config-volume\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:38.574504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.574458    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4sgv\" (UniqueName: \"kubernetes.io/projected/f5c20692-0f49-4d2c-9a87-a672e7cc6e87-kube-api-access-d4sgv\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:38.574504 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.574471    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjkq\" (UniqueName: \"kubernetes.io/projected/c920d708-0527-417f-88c0-c10fa9a7bb0b-kube-api-access-kzjkq\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:38.665591 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.665554    5282 generic.go:356] "Generic (PLEG): container finished" podID="c920d708-0527-417f-88c0-c10fa9a7bb0b" containerID="403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa" exitCode=0
Nov 03 20:47:38.665765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.665604    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"
Nov 03 20:47:38.665765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.665643    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt" event={"ID":"c920d708-0527-417f-88c0-c10fa9a7bb0b","Type":"ContainerDied","Data":"403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa"}
Nov 03 20:47:38.665765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.665692    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt" event={"ID":"c920d708-0527-417f-88c0-c10fa9a7bb0b","Type":"ContainerDied","Data":"600d1a8b948af4c165a5286d2dfae901ac4560ba75f35c438e4c716dc50c8bf3"}
Nov 03 20:47:38.665765 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.665715    5282 scope.go:122] "RemoveContainer" containerID="403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa"
Nov 03 20:47:38.666910 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.666885    5282 generic.go:356] "Generic (PLEG): container finished" podID="f5c20692-0f49-4d2c-9a87-a672e7cc6e87" containerID="b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28" exitCode=0
Nov 03 20:47:38.667040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.666920    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"
Nov 03 20:47:38.667040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.666952    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq" event={"ID":"f5c20692-0f49-4d2c-9a87-a672e7cc6e87","Type":"ContainerDied","Data":"b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28"}
Nov 03 20:47:38.667040 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.666983    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq" event={"ID":"f5c20692-0f49-4d2c-9a87-a672e7cc6e87","Type":"ContainerDied","Data":"4568dfe69b8af6a58538a19e7b1a27bc293fc71edf72fb44f012a64ddb08a547"}
Nov 03 20:47:38.676294 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.676275    5282 scope.go:122] "RemoveContainer" containerID="403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa"
Nov 03 20:47:38.676564 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:47:38.676544    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa\": container with ID starting with 403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa not found: ID does not exist" containerID="403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa"
Nov 03 20:47:38.676705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.676572    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa"} err="failed to get container status \"403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa\": rpc error: code = NotFound desc = could not find container \"403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa\": container with ID starting with 403ebc097aeb625699c74b598c9c351410477834bae3309140bb3f31566547aa not found: ID does not exist"
Nov 03 20:47:38.676705 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.676588    5282 scope.go:122] "RemoveContainer" containerID="b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28"
Nov 03 20:47:38.685613 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.685590    5282 scope.go:122] "RemoveContainer" containerID="b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28"
Nov 03 20:47:38.685913 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:47:38.685894    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28\": container with ID starting with b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28 not found: ID does not exist" containerID="b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28"
Nov 03 20:47:38.685996 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.685919    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28"} err="failed to get container status \"b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28\": rpc error: code = NotFound desc = could not find container \"b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28\": container with ID starting with b2ce6e49390eb977f7e458457c92831f00c30de646996e7adc969c81f0ea5d28 not found: ID does not exist"
Nov 03 20:47:38.690709 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.690686    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"]
Nov 03 20:47:38.695813 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.695790    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-ca53c0-866db7d97f-6qvzt"]
Nov 03 20:47:38.705800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.705777    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"]
Nov 03 20:47:38.711227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.711186    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-ca53c0-7777855786-s9mkq"]
Nov 03 20:47:38.964745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:38.964663    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:39.453181 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:39.453145    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c920d708-0527-417f-88c0-c10fa9a7bb0b" path="/var/lib/kubelet/pods/c920d708-0527-417f-88c0-c10fa9a7bb0b/volumes"
Nov 03 20:47:39.453578 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:39.453539    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c20692-0f49-4d2c-9a87-a672e7cc6e87" path="/var/lib/kubelet/pods/f5c20692-0f49-4d2c-9a87-a672e7cc6e87/volumes"
Nov 03 20:47:39.669944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:39.669912    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:47:45.113430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:45.113388    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:45.252695 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:45.252663    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:45.373799 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:45.373711    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"]
Nov 03 20:47:45.373969 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:45.373945    5282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy" containerID="cri-o://f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89" gracePeriod=30
Nov 03 20:47:45.557465 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:45.557433    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:47.464890 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:47.464856    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:47:49.958413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:49.958328    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-fj62v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:47:49.958413 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:49.958384    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-fj62v" podUID="39a446f2-8afb-4a18-9625-ef1c817a0883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:47:49.993321 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:49.993288    5282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-2cl2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:47:49.993618 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:49.993575    5282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-2cl2z" podUID="1f0af13b-f24a-4817-a5da-a4f9ea8402ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:47:50.616575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.616546    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:47:50.635346 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635316    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-socket\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635354    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-credential-socket\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635378    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-token\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-token\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635428    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-kube-api-access-966qf\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-kube-api-access-966qf\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635488 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635478    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-data\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635505    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istiod-ca-cert\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635561    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-envoy\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635620    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-podinfo\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.635739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635657    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-credential-socket" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:47:50.635739 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635681    5282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-certs\") pod \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\" (UID: \"26f0fe09-4261-4779-af7b-10ccf9dd1bf4\") "
Nov 03 20:47:50.636019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635817    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-data" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:47:50.636019 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.635905    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istiod-ca-cert" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:50.636141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.636032    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-certs" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:47:50.636141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.636041    5282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-data\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.636141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.636064    5282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istiod-ca-cert\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.636141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.636080    5282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-credential-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.636141 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.636101    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-socket" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:47:50.638559 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.638534    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-token" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:50.638745 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.638723    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-envoy" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:47:50.639430 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.639378    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-podinfo" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:47:50.639575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.639528    5282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-kube-api-access-966qf" pod "26f0fe09-4261-4779-af7b-10ccf9dd1bf4" (UID: "26f0fe09-4261-4779-af7b-10ccf9dd1bf4"). InnerVolumeSpecName "kube-api-access-966qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:50.736839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.736796    5282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966qf\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-kube-api-access-966qf\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.736839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.736832    5282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-envoy\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.736839 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.736846    5282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-podinfo\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.737090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.736860    5282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-certs\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.737090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.736872    5282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-workload-socket\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.737090 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.736883    5282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f0fe09-4261-4779-af7b-10ccf9dd1bf4-istio-token\") on node \"ip-10-0-142-244.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:50.748179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.748145    5282 generic.go:356] "Generic (PLEG): container finished" podID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerID="f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89" exitCode=0
Nov 03 20:47:50.748357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.748190    5282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"
Nov 03 20:47:50.748357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.748226    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" event={"ID":"26f0fe09-4261-4779-af7b-10ccf9dd1bf4","Type":"ContainerDied","Data":"f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89"}
Nov 03 20:47:50.748357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.748293    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f" event={"ID":"26f0fe09-4261-4779-af7b-10ccf9dd1bf4","Type":"ContainerDied","Data":"e92b3aefdf3f4e92a38818f52197c22c8a3507f6969ebed009a10bf3d8b1bca4"}
Nov 03 20:47:50.748357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.748316    5282 scope.go:122] "RemoveContainer" containerID="f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89"
Nov 03 20:47:50.757828 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.757808    5282 scope.go:122] "RemoveContainer" containerID="bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234"
Nov 03 20:47:50.765633 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.765608    5282 scope.go:122] "RemoveContainer" containerID="f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89"
Nov 03 20:47:50.765892 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:47:50.765873    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89\": container with ID starting with f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89 not found: ID does not exist" containerID="f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89"
Nov 03 20:47:50.765939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.765902    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89"} err="failed to get container status \"f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89\": rpc error: code = NotFound desc = could not find container \"f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89\": container with ID starting with f58fc5eb909a4d87455eee2ba60081fd7301a03f793401b9fc81a047efb75d89 not found: ID does not exist"
Nov 03 20:47:50.765939 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.765920    5282 scope.go:122] "RemoveContainer" containerID="bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234"
Nov 03 20:47:50.766155 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:47:50.766136    5282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234\": container with ID starting with bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234 not found: ID does not exist" containerID="bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234"
Nov 03 20:47:50.766224 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.766164    5282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234"} err="failed to get container status \"bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234\": rpc error: code = NotFound desc = could not find container \"bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234\": container with ID starting with bd6ea6db7516a77194620413008d36fd0688e70a9125d7250b1b946e0122a234 not found: ID does not exist"
Nov 03 20:47:50.769500 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.769476    5282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"]
Nov 03 20:47:50.773437 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:50.773416    5282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ca53c0-openshift-default-cb7547f57-pdg5f"]
Nov 03 20:47:51.453167 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:51.453136    5282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" path="/var/lib/kubelet/pods/26f0fe09-4261-4779-af7b-10ccf9dd1bf4/volumes"
Nov 03 20:47:52.575216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:52.575169    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:47:59.009226 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:59.009177    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:47:59.136386 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:47:59.136349    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:01.612755 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:48:01.612718    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:48:07.670304 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:07.670273    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:48:23.120176 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:23.120141    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:24.736428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:24.736391    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:48:31.613271 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:48:31.613231    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:48:34.271021 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:34.270985    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-lm5fq"]
Nov 03 20:48:45.187534 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:45.187505    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rrtn2_8bfd6f63-1e63-426c-9728-c3de74713bf1/manager/0.log"
Nov 03 20:48:45.837008 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:45.836973    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f5dc4b79-ptlhp_db3f2c57-f26a-48ce-bb62-ce619f8d622f/manager/0.log"
Nov 03 20:48:46.049880 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:46.049853    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bbs7v_6887d4ff-a9ab-423f-b73b-f681f1e8e474/server/0.log"
Nov 03 20:48:47.359355 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:47.359317    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-876c878b5-njv6s_324d125b-1199-4c96-ba01-2928290bd3f8/maas-api/0.log"
Nov 03 20:48:50.701842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:50.701811    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fbz8b_6beed4ed-3890-4845-952a-71bb8ed502bf/kuadrant-console-plugin/0.log"
Nov 03 20:48:51.113300 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:51.113270    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lm5fq_c4159e7e-3404-4698-9fba-aea080862989/limitador/0.log"
Nov 03 20:48:52.315010 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:52.314977    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-85svm_860c0078-098f-46a0-832d-507c6dc76dc8/discovery/0.log"
Nov 03 20:48:52.674185 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:52.674103    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-j2xmd_31f3f7bc-4ef1-43fb-83ff-36926bf67807/kube-auth-proxy/0.log"
Nov 03 20:48:52.853230 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:52.853179    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2cl2z_1f0af13b-f24a-4817-a5da-a4f9ea8402ac/istio-proxy/1.log"
Nov 03 20:48:53.599934 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:53.599903    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-fj62v_39a446f2-8afb-4a18-9625-ef1c817a0883/istio-proxy/1.log"
Nov 03 20:48:54.127392 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:54.127364    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-krhrv_f0ac7456-d600-48bd-b83d-8ed330bf70c0/istio-proxy/0.log"
Nov 03 20:48:54.818608 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:54.818581    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qg778_a267af30-6705-4247-a544-4c799fa2326b/payload-pre-processing/0.log"
Nov 03 20:48:54.985107 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:54.985075    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:48:55.571018 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:55.570986    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2cl2z_1f0af13b-f24a-4817-a5da-a4f9ea8402ac/istio-proxy/0.log"
Nov 03 20:48:55.936012 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:55.935928    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-fj62v_39a446f2-8afb-4a18-9625-ef1c817a0883/istio-proxy/0.log"
Nov 03 20:48:56.992456 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:56.992424    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b_95213a91-c3dd-4195-bcea-9e329e10679b/main/0.log"
Nov 03 20:48:57.491289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:57.491254    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-44wlb_3bf6a00b-f4b7-4909-8155-e849fecd35ad/main/0.log"
Nov 03 20:48:58.035133 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:48:58.035087    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl_04e563bf-735c-4d94-9de1-d04994490f4b/main/0.log"
Nov 03 20:49:01.612980 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:49:01.612937    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:49:08.328030 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:08.327991    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rrtn2_8bfd6f63-1e63-426c-9728-c3de74713bf1/manager/0.log"
Nov 03 20:49:08.955147 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:08.955111    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f5dc4b79-ptlhp_db3f2c57-f26a-48ce-bb62-ce619f8d622f/manager/0.log"
Nov 03 20:49:09.163245 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:09.163187    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bbs7v_6887d4ff-a9ab-423f-b73b-f681f1e8e474/server/0.log"
Nov 03 20:49:10.273558 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:10.273526    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-876c878b5-njv6s_324d125b-1199-4c96-ba01-2928290bd3f8/maas-api/0.log"
Nov 03 20:49:13.162594 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:13.162567    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fbz8b_6beed4ed-3890-4845-952a-71bb8ed502bf/kuadrant-console-plugin/0.log"
Nov 03 20:49:13.549045 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:13.549007    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lm5fq_c4159e7e-3404-4698-9fba-aea080862989/limitador/0.log"
Nov 03 20:49:14.448915 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:14.448884    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-85svm_860c0078-098f-46a0-832d-507c6dc76dc8/discovery/0.log"
Nov 03 20:49:14.803662 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:14.803629    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-j2xmd_31f3f7bc-4ef1-43fb-83ff-36926bf67807/kube-auth-proxy/0.log"
Nov 03 20:49:14.983684 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:14.983616    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2cl2z_1f0af13b-f24a-4817-a5da-a4f9ea8402ac/istio-proxy/1.log"
Nov 03 20:49:15.691138 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:15.691108    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-fj62v_39a446f2-8afb-4a18-9625-ef1c817a0883/istio-proxy/1.log"
Nov 03 20:49:16.215036 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:16.214987    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-krhrv_f0ac7456-d600-48bd-b83d-8ed330bf70c0/istio-proxy/0.log"
Nov 03 20:49:16.927691 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:16.927663    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qg778_a267af30-6705-4247-a544-4c799fa2326b/payload-pre-processing/0.log"
Nov 03 20:49:17.107930 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:17.107876    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:49:17.828385 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:17.828352    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-z2t2b_95213a91-c3dd-4195-bcea-9e329e10679b/main/0.log"
Nov 03 20:49:18.354188 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:18.354159    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-44wlb_3bf6a00b-f4b7-4909-8155-e849fecd35ad/main/0.log"
Nov 03 20:49:18.877039 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:18.877000    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rrvkl_04e563bf-735c-4d94-9de1-d04994490f4b/main/0.log"
Nov 03 20:49:23.158314 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:23.158274    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:49:31.613821 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:49:31.613782    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:49:56.563611 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:56.563579    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f5dc4b79-ptlhp_db3f2c57-f26a-48ce-bb62-ce619f8d622f/manager/0.log"
Nov 03 20:49:56.838408 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:56.838324    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f5dc4b79-ptlhp_db3f2c57-f26a-48ce-bb62-ce619f8d622f/manager/0.log"
Nov 03 20:49:57.552843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:57.552816    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:49:57.728053 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:49:57.728024    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:50:01.613486 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:50:01.613436    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:50:22.123843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:22.123806    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:23.215120 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:23.215084    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:30.350240 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:30.350185    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:31.613643 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:50:31.613604    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:50:42.869422 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:42.869374    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f5dc4b79-ptlhp_db3f2c57-f26a-48ce-bb62-ce619f8d622f/manager/0.log"
Nov 03 20:50:43.130854 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:43.130770    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65f5dc4b79-ptlhp_db3f2c57-f26a-48ce-bb62-ce619f8d622f/manager/0.log"
Nov 03 20:50:43.810649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:43.810616    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:50:43.971905 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:43.971873    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:50:44.297720 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.297677    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pd2z/must-gather-q68tc"]
Nov 03 20:50:44.298142 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298122    5282 cpu_manager.go:414] "containerMap: removing container" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy"
Nov 03 20:50:44.298254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298143    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy"
Nov 03 20:50:44.298254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298165    5282 cpu_manager.go:414] "containerMap: removing container" podUID="c920d708-0527-417f-88c0-c10fa9a7bb0b" containerName="payload-pre-processing"
Nov 03 20:50:44.298254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298174    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c920d708-0527-417f-88c0-c10fa9a7bb0b" containerName="payload-pre-processing"
Nov 03 20:50:44.298254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298216    5282 cpu_manager.go:414] "containerMap: removing container" podUID="f5c20692-0f49-4d2c-9a87-a672e7cc6e87" containerName="payload-processing"
Nov 03 20:50:44.298254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298225    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c20692-0f49-4d2c-9a87-a672e7cc6e87" containerName="payload-processing"
Nov 03 20:50:44.298254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298240    5282 cpu_manager.go:414] "containerMap: removing container" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy"
Nov 03 20:50:44.298254 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298247    5282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy"
Nov 03 20:50:44.298649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298378    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy"
Nov 03 20:50:44.298649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298399    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f0fe09-4261-4779-af7b-10ccf9dd1bf4" containerName="istio-proxy"
Nov 03 20:50:44.298649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298412    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c920d708-0527-417f-88c0-c10fa9a7bb0b" containerName="payload-pre-processing"
Nov 03 20:50:44.298649 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.298423    5282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c20692-0f49-4d2c-9a87-a672e7cc6e87" containerName="payload-processing"
Nov 03 20:50:44.301774 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.301751    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.306436 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.306415    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pd2z\"/\"openshift-service-ca.crt\""
Nov 03 20:50:44.306556 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.306492    5282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5pd2z\"/\"kube-root-ca.crt\""
Nov 03 20:50:44.307502 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.307484    5282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5pd2z\"/\"default-dockercfg-pvhgd\""
Nov 03 20:50:44.359483 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.359445    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/must-gather-q68tc"]
Nov 03 20:50:44.363066 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.363044    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:44.380727 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.380699    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgdt\" (UniqueName: \"kubernetes.io/projected/5f7b742d-fdd7-43a3-b0b4-5674e681311a-kube-api-access-qpgdt\") pod \"must-gather-q68tc\" (UID: \"5f7b742d-fdd7-43a3-b0b4-5674e681311a\") " pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.380843 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.380748    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5f7b742d-fdd7-43a3-b0b4-5674e681311a-must-gather-output\") pod \"must-gather-q68tc\" (UID: \"5f7b742d-fdd7-43a3-b0b4-5674e681311a\") " pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.481627 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.481592    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgdt\" (UniqueName: \"kubernetes.io/projected/5f7b742d-fdd7-43a3-b0b4-5674e681311a-kube-api-access-qpgdt\") pod \"must-gather-q68tc\" (UID: \"5f7b742d-fdd7-43a3-b0b4-5674e681311a\") " pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.481808 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.481640    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5f7b742d-fdd7-43a3-b0b4-5674e681311a-must-gather-output\") pod \"must-gather-q68tc\" (UID: \"5f7b742d-fdd7-43a3-b0b4-5674e681311a\") " pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.481997 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.481980    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5f7b742d-fdd7-43a3-b0b4-5674e681311a-must-gather-output\") pod \"must-gather-q68tc\" (UID: \"5f7b742d-fdd7-43a3-b0b4-5674e681311a\") " pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.499714 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.499692    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgdt\" (UniqueName: \"kubernetes.io/projected/5f7b742d-fdd7-43a3-b0b4-5674e681311a-kube-api-access-qpgdt\") pod \"must-gather-q68tc\" (UID: \"5f7b742d-fdd7-43a3-b0b4-5674e681311a\") " pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.615306 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.615219    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/must-gather-q68tc"
Nov 03 20:50:44.885038 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.884967    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/must-gather-q68tc"]
Nov 03 20:50:44.887600 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:50:44.887569    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f7b742d_fdd7_43a3_b0b4_5674e681311a.slice/crio-405b691caf0ba9e387478ce0602489713e7c9ebe9bcba23918d7f05ecf2555aa WatchSource:0}: Error finding container 405b691caf0ba9e387478ce0602489713e7c9ebe9bcba23918d7f05ecf2555aa: Status 404 returned error can't find the container with id 405b691caf0ba9e387478ce0602489713e7c9ebe9bcba23918d7f05ecf2555aa
Nov 03 20:50:44.889364 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:44.889345    5282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:50:45.549280 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:45.549234    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/must-gather-q68tc" event={"ID":"5f7b742d-fdd7-43a3-b0b4-5674e681311a","Type":"ContainerStarted","Data":"405b691caf0ba9e387478ce0602489713e7c9ebe9bcba23918d7f05ecf2555aa"}
Nov 03 20:50:46.404006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:46.402608    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:50:46.556006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:46.555964    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/must-gather-q68tc" event={"ID":"5f7b742d-fdd7-43a3-b0b4-5674e681311a","Type":"ContainerStarted","Data":"1b071e1ccc05d902be5a629ff7f7467a7a2388157c3f75280a527221b2326e7d"}
Nov 03 20:50:46.556006 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:46.556012    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/must-gather-q68tc" event={"ID":"5f7b742d-fdd7-43a3-b0b4-5674e681311a","Type":"ContainerStarted","Data":"453e4c0a733b2dd55ba923028d843cfd9d1d5b8efbd86e6196e35dbf3c671775"}
Nov 03 20:50:46.572125 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:46.572074    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5pd2z/must-gather-q68tc" podStartSLOduration=1.834274181 podStartE2EDuration="2.572058873s" totalImagesPullingTime="737.784692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:46.570372026 +0000 UTC m=+4239.239573501" watchObservedRunningTime="2026-11-03 20:50:46.572058873 +0000 UTC m=+4239.241260381"
Nov 03 20:50:47.359116 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:47.359078    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2g52t_6b3aec1c-dcd1-4b89-86cc-898550befdad/global-pull-secret-syncer/0.log"
Nov 03 20:50:47.626881 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:47.626793    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log"
Nov 03 20:50:47.812062 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:47.812021    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-244.us-east-2.compute.internal_548ab8487a64671c8f29d89cabd4d700/haproxy/0.log"
Nov 03 20:50:54.826227 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:54.826171    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fbz8b_6beed4ed-3890-4845-952a-71bb8ed502bf/kuadrant-console-plugin/0.log"
Nov 03 20:50:54.943216 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:54.942555    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-lm5fq_c4159e7e-3404-4698-9fba-aea080862989/limitador/0.log"
Nov 03 20:50:56.573884 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.573837    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-z2zd5_58e236c2-f2bc-4adc-8daf-f160c4c882f1/monitoring-plugin/0.log"
Nov 03 20:50:56.693584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.693553    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b2qlc_48f77550-ff58-46e6-b7d8-6840e4afa52c/node-exporter/0.log"
Nov 03 20:50:56.716455 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.716430    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b2qlc_48f77550-ff58-46e6-b7d8-6840e4afa52c/kube-rbac-proxy/0.log"
Nov 03 20:50:56.751853 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.751818    5282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"]
Nov 03 20:50:56.755840 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.755809    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b2qlc_48f77550-ff58-46e6-b7d8-6840e4afa52c/init-textfile/0.log"
Nov 03 20:50:56.768114 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.768083    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"]
Nov 03 20:50:56.768325 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.768224    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:56.945632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.945548    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-sys\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:56.945632 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.945621    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99bq\" (UniqueName: \"kubernetes.io/projected/fb01e24b-a7d2-4344-b08a-6894425e5340-kube-api-access-n99bq\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:56.945826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.945747    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-proc\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:56.945826 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.945804    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-podres\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:56.945944 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:56.945923    5282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-lib-modules\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.048737 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.048693    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-lib-modules\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.049042 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.049021    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-sys\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.049179 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.049164    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99bq\" (UniqueName: \"kubernetes.io/projected/fb01e24b-a7d2-4344-b08a-6894425e5340-kube-api-access-n99bq\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.049338 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.049323    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-proc\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.049447 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.049436    5282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-podres\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.049645 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.049631    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-podres\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.049802 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.049788    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-lib-modules\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.049926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.049911    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-sys\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.050323 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.050293    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fb01e24b-a7d2-4344-b08a-6894425e5340-proc\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.057810 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.057786    5282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99bq\" (UniqueName: \"kubernetes.io/projected/fb01e24b-a7d2-4344-b08a-6894425e5340-kube-api-access-n99bq\") pod \"perf-node-gather-daemonset-2qhs6\" (UID: \"fb01e24b-a7d2-4344-b08a-6894425e5340\") " pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.086525 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.086503    5282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.141740 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.141708    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ztz7h_076c5bac-22d5-4145-9dd6-bfc03e1b858e/prometheus-operator/0.log"
Nov 03 20:50:57.164912 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.164850    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ztz7h_076c5bac-22d5-4145-9dd6-bfc03e1b858e/kube-rbac-proxy/0.log"
Nov 03 20:50:57.227769 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.227706    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59cf774bd5-lp2nw_dcbb4036-9624-4a3f-b866-af74b917bd07/telemeter-client/0.log"
Nov 03 20:50:57.250846 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.250793    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59cf774bd5-lp2nw_dcbb4036-9624-4a3f-b866-af74b917bd07/reload/0.log"
Nov 03 20:50:57.274780 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.274748    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59cf774bd5-lp2nw_dcbb4036-9624-4a3f-b866-af74b917bd07/kube-rbac-proxy/0.log"
Nov 03 20:50:57.304085 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.304055    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8cd547dc-9csvs_9a113235-00b3-42db-ac16-0fc01da5a788/thanos-query/0.log"
Nov 03 20:50:57.325994 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.325961    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8cd547dc-9csvs_9a113235-00b3-42db-ac16-0fc01da5a788/kube-rbac-proxy-web/0.log"
Nov 03 20:50:57.351971 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.351936    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8cd547dc-9csvs_9a113235-00b3-42db-ac16-0fc01da5a788/kube-rbac-proxy/0.log"
Nov 03 20:50:57.370440 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.370056    5282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"]
Nov 03 20:50:57.374875 ip-10-0-142-244 kubenswrapper[5282]: W1103 20:50:57.374843    5282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfb01e24b_a7d2_4344_b08a_6894425e5340.slice/crio-f4982a4e1e47f72d5afe21a75391a041fddfdfbee058afba4a3fbd557ff284cc WatchSource:0}: Error finding container f4982a4e1e47f72d5afe21a75391a041fddfdfbee058afba4a3fbd557ff284cc: Status 404 returned error can't find the container with id f4982a4e1e47f72d5afe21a75391a041fddfdfbee058afba4a3fbd557ff284cc
Nov 03 20:50:57.375428 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.375409    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8cd547dc-9csvs_9a113235-00b3-42db-ac16-0fc01da5a788/prom-label-proxy/0.log"
Nov 03 20:50:57.411926 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.411903    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8cd547dc-9csvs_9a113235-00b3-42db-ac16-0fc01da5a788/kube-rbac-proxy-rules/0.log"
Nov 03 20:50:57.435842 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.435815    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8cd547dc-9csvs_9a113235-00b3-42db-ac16-0fc01da5a788/kube-rbac-proxy-metrics/0.log"
Nov 03 20:50:57.658832 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.658800    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6" event={"ID":"fb01e24b-a7d2-4344-b08a-6894425e5340","Type":"ContainerStarted","Data":"0d17069be8122adea6f0afa5a80cc2addd18ec846d1887583a83812a2b65bd26"}
Nov 03 20:50:57.659229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.658854    5282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6" event={"ID":"fb01e24b-a7d2-4344-b08a-6894425e5340","Type":"ContainerStarted","Data":"f4982a4e1e47f72d5afe21a75391a041fddfdfbee058afba4a3fbd557ff284cc"}
Nov 03 20:50:57.659229 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.658878    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:50:57.674246 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:50:57.674175    5282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6" podStartSLOduration=1.674160349 podStartE2EDuration="1.674160349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:57.673186491 +0000 UTC m=+4250.342387961" watchObservedRunningTime="2026-11-03 20:50:57.674160349 +0000 UTC m=+4250.343361815"
Nov 03 20:51:01.544804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:01.544677    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fqbg_43f9158b-9674-477d-a64f-f19fe00bed5b/dns/0.log"
Nov 03 20:51:01.567075 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:01.567053    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fqbg_43f9158b-9674-477d-a64f-f19fe00bed5b/kube-rbac-proxy/0.log"
Nov 03 20:51:01.615854 ip-10-0-142-244 kubenswrapper[5282]: E1103 20:51:01.615819    5282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bv4d2" containerName="extractor"
Nov 03 20:51:01.700641 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:01.700610    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjqb6_ce3c36fc-764d-4dc1-a515-000a6407542a/dns-node-resolver/0.log"
Nov 03 20:51:02.481577 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:02.480357    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-km679_e5ef855a-3c98-47fd-8559-bb40ad4db198/node-ca/0.log"
Nov 03 20:51:03.683289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:03.683255    5282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5pd2z/perf-node-gather-daemonset-2qhs6"
Nov 03 20:51:05.130584 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:05.130556    5282 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dlgxk_0fc369cd-d32a-4cad-9da6-d5bfd899591a/konnectivity-agent/0.log" limit=1024
Nov 03 20:51:05.267014 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:05.266978    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-85svm_860c0078-098f-46a0-832d-507c6dc76dc8/discovery/0.log"
Nov 03 20:51:05.782071 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:05.782036    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-j2xmd_31f3f7bc-4ef1-43fb-83ff-36926bf67807/kube-auth-proxy/0.log"
Nov 03 20:51:05.820357 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:05.820328    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2cl2z_1f0af13b-f24a-4817-a5da-a4f9ea8402ac/istio-proxy/1.log"
Nov 03 20:51:05.820575 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:05.820565    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2cl2z_1f0af13b-f24a-4817-a5da-a4f9ea8402ac/istio-proxy/0.log"
Nov 03 20:51:05.941755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:05.941728    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-fj62v_39a446f2-8afb-4a18-9625-ef1c817a0883/istio-proxy/1.log"
Nov 03 20:51:05.941755 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:05.941752    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-fj62v_39a446f2-8afb-4a18-9625-ef1c817a0883/istio-proxy/0.log"
Nov 03 20:51:06.025311 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:06.025276    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-krhrv_f0ac7456-d600-48bd-b83d-8ed330bf70c0/istio-proxy/0.log"
Nov 03 20:51:06.130289 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:06.130218    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qg778_a267af30-6705-4247-a544-4c799fa2326b/payload-pre-processing/0.log"
Nov 03 20:51:06.158113 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:06.158079    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:51:08.727514 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:08.727384    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-st6kb_6bb8d09d-f69e-4920-a257-0c957bebd264/serve-healthcheck-canary/0.log"
Nov 03 20:51:09.373812 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.373776    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bv4d2_a9af4d64-b784-4e51-aa42-038905055202/kube-rbac-proxy/0.log"
Nov 03 20:51:09.400493 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.400465    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bv4d2_a9af4d64-b784-4e51-aa42-038905055202/exporter/0.log"
Nov 03 20:51:09.426553 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.426518    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bv4d2_a9af4d64-b784-4e51-aa42-038905055202/extractor/0.log"
Nov 03 20:51:09.557248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.557180    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-85svm_860c0078-098f-46a0-832d-507c6dc76dc8/discovery/0.log"
Nov 03 20:51:09.585804 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.585770    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dd88dbbb7-j2xmd_31f3f7bc-4ef1-43fb-83ff-36926bf67807/kube-auth-proxy/0.log"
Nov 03 20:51:09.607827 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.607782    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2cl2z_1f0af13b-f24a-4817-a5da-a4f9ea8402ac/istio-proxy/1.log"
Nov 03 20:51:09.608317 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.608293    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-2cl2z_1f0af13b-f24a-4817-a5da-a4f9ea8402ac/istio-proxy/0.log"
Nov 03 20:51:09.681266 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.681185    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-fj62v_39a446f2-8afb-4a18-9625-ef1c817a0883/istio-proxy/1.log"
Nov 03 20:51:09.681498 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.681480    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-fj62v_39a446f2-8afb-4a18-9625-ef1c817a0883/istio-proxy/0.log"
Nov 03 20:51:09.727485 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.727450    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-krhrv_f0ac7456-d600-48bd-b83d-8ed330bf70c0/istio-proxy/0.log"
Nov 03 20:51:09.790929 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.790896    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qg778_a267af30-6705-4247-a544-4c799fa2326b/payload-pre-processing/0.log"
Nov 03 20:51:09.807800 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:09.807758    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-dj76x_82bd2aaa-93d7-415c-aabe-b77f54e6f3be/payload-processing/0.log"
Nov 03 20:51:11.985218 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:11.984651    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5749d45fbb-kqx6s_7e296f90-4f55-42f9-9310-f31d1112881e/manager/0.log"
Nov 03 20:51:12.003248 ip-10-0-142-244 kubenswrapper[5282]: I1103 20:51:12.003221    5282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5749d45fbb-kqx6s_7e296f90-4f55-42f9-9310-f31d1112881e/copy-manifests/0.log"